Devices, Methods, and Graphical User Interfaces for Displaying Objects in 3D Context

ABSTRACT

An electronic device displays an environment that includes a virtual object that is associated with a first action that is triggered based on satisfaction of a first set of criteria. In response to a first input detected while the environment is displayed: the first action is performed in accordance with a determination that the first input satisfies the first set of criteria, and, in accordance with a determination that the first input does not satisfy the first set of criteria but instead satisfies a second set of criteria, the first action is not performed and instead a first visual indication of one or more inputs that if performed would cause the first set of criteria to be satisfied is displayed.

RELATED APPLICATIONS

This application claims priority to U.S. Provisional Application Ser.No. 62/855,973, filed Jun. 1, 2019 and U.S. Provisional Application Ser.No. 62/844,010, filed May 6, 2019, which are incorporated by referenceherein in their entireties.

This application is related to U.S. Provisional Application Ser. No.62/679,951, filed Jun. 3, 2018 and U.S. Provisional Application Ser. No.62/621,529, filed Jan. 24, 2018, which are incorporated by referenceherein in their entireties.

TECHNICAL FIELD

This relates generally to electronic devices that display virtualobjects, including but not limited to electronic devices that displayvirtual objects in a variety of contexts.

BACKGROUND

The development of computer systems for augmented reality has increasedsignificantly in recent years. Example augmented reality environmentsinclude at least some virtual elements that replace or augment thephysical world. Input devices, such as touch-sensitive surfaces, forcomputer systems and other electronic computing devices are used tointeract with virtual/augmented reality environments. Exampletouch-sensitive surfaces include touchpads, touch-sensitive remotecontrols, and touch-screen displays. Such surfaces are used tomanipulate user interfaces and objects therein on a display. Exampleuser interface objects include digital images, video, text, icons, andcontrol elements such as buttons and other graphics.

But methods and interfaces for interacting with environments thatinclude at least some virtual elements (e.g., applications, augmentedreality environments, mixed reality environments, and virtual realityenvironments) are cumbersome, inefficient, and limited. For example,systems that provide insufficient feedback for performing actionsassociated with virtual objects, systems that require a series of inputsto generate virtual objects suitable for display in an augmented realityenvironment, and systems that require lack handling for manipulatingsets of virtual objects are tedious, create a significant cognitiveburden on a user, and detract from the experience with thevirtual/augmented reality environment. In addition, these methods takelonger than necessary, thereby wasting energy. This latter considerationis particularly important in battery-operated devices.

SUMMARY

Accordingly, there is a need for computer systems with improved methodsand interfaces for interacting with virtual objects. Such methods andinterfaces optionally complement or replace conventional methods forinteracting with virtual objects. Such methods and interfaces reduce thenumber, extent, and/or nature of the inputs from a user and produce amore efficient human-machine interface. For battery-operated devices,such methods and interfaces conserve power and increase the time betweenbattery charges.

The above deficiencies and other problems associated with interfaces forinteracting with virtual objects (e.g., user interfaces for augmentedreality (AR) and related non-AR interfaces) are reduced or eliminated bythe disclosed computer systems. In some embodiments, the computer systemincludes a desktop computer. In some embodiments, the computer system isportable (e.g., a notebook computer, tablet computer, or handhelddevice). In some embodiments, the computer system includes a personalelectronic device (e.g., a wearable electronic device, such as a watch).In some embodiments, the computer system has (and/or is in communicationwith) a touchpad. In some embodiments, the computer system has (and/oris in communication with) a touch-sensitive display (also known as a“touch screen” or “touch-screen display”). In some embodiments, thecomputer system has a graphical user interface (GUI), one or moreprocessors, memory and one or more modules, programs or sets ofinstructions stored in the memory for performing multiple functions. Insome embodiments, the user interacts with the GUI in part through stylusand/or finger contacts and gestures on the touch-sensitive surface. Insome embodiments, the functions optionally include game playing, imageediting, drawing, presenting, word processing, spreadsheet making,telephoning, video conferencing, e-mailing, instant messaging, workoutsupport, digital photographing, digital videoing, web browsing, digitalmusic playing, note taking, and/or digital video playing. Executableinstructions for performing these functions are, optionally, included ina non-transitory computer readable storage medium or other computerprogram product configured for execution by one or more processors.

In accordance with some embodiments, a method is performed at anelectronic device including a display generation component and one ormore input devices. The method includes displaying, via the displaygeneration component, at least a portion of an environment that includesa virtual object that is associated with a first action that istriggered based on satisfaction of a first set of criteria. The methodfurther includes, while displaying, by the display generation component,the portion of the environment, detecting a first input. The methodfurther includes, in response to detecting the first input: inaccordance with a determination that the first input satisfies the firstset of criteria, performing the first action; and in accordance with adetermination that the first input does not satisfy the first set ofcriteria but instead satisfies a second set of criteria, forgoingperforming the first action and instead displaying a first visualindication of one or more inputs that if performed would cause the firstset of criteria to be satisfied.

In accordance with some embodiments, a method is performed at anelectronic device including a display generation component, one or moreinput devices, and one or more cameras. The method includes displaying,by the display generation component: a first user interface region, afirst representation of a first media item in the first user interfaceregion, and a first user interface object for displaying a sharing userinterface. The method further includes, while displaying the first userinterface region, detecting a first input corresponding to the sharinguser interface object. The method further includes, in response todetecting the first input corresponding to the first user interfaceobject, displaying the sharing user interface that includes a seconduser interface object for initiating a process for displaying a secondrepresentation of the first media item in an augmented realityenvironment. The method further includes, while the sharing userinterface is displayed, detecting a sequence of one or more inputsincluding selection of the second user interface object. The methodfurther includes, in response to detecting the sequence of one or moreinputs including selection of the second user interface object,displaying the second representation of the first media item in a seconduser interface that includes content of at least a portion of the fieldof view of the one or more cameras.

In accordance with some embodiments, a method is performed at anelectronic device including a display generation component, one or moreinput devices, and one or more cameras. The method includes displaying,by the display generation component, a plurality of virtual objects. Themethod further includes, while displaying the plurality of virtualobjects, detecting a first manipulation input directed to a respectivevirtual object of the plurality of virtual objects. The method furtherincludes, in response to detecting the first manipulation input directedto the respective virtual object of the plurality of virtual objects: inaccordance with a determination that the plurality of virtual objectsare displayed with at least a portion of a field of view of the one ormore cameras, changing a display property of the respective virtualobject based on the first manipulation input; and, in accordance with adetermination that the plurality of virtual objects are displayed in auser interface that does not include the field of view of the one ormore cameras, changing a display property of the plurality of virtualobjects based on the first manipulation input.

In accordance with some embodiments, a method is performed at anelectronic device including a display generation component, one or moreinput devices, and one or more cameras. The method includes displaying,by the display generation component, a representation of a media item.The method further includes detecting an input corresponding to arequest to display the media item in an augmented reality environment.The method further includes, in response to detecting the inputcorresponding to a request to display the media item in an augmentedreality environment: in accordance with a determination that the mediaitem has a property that does not meet compatibility criteria fordisplay in the augmented reality environment, displaying a prompt tochange the property of the media item; and in accordance with adetermination that the media item meets the compatibility criteria,displaying a virtual object that corresponds to the media item withcontent of at least a portion of the field of view of the one or morecameras.

In accordance with some embodiments, a method is performed at anelectronic device including a display generation component and one ormore input devices. The method includes displaying, via the displaygeneration component, a first user interface with a first representationof content. The method further includes, while displaying the first userinterface with the first representation of the content, receiving, viathe one or more input devices, a request to display a virtual model thatcorresponds to the content. The method further includes, in response toreceiving the request to display the virtual model that corresponds tothe content: in accordance with a determination that the first userinterface is configured to perform an operation associated with thecontent, displaying the virtual model of the content concurrently with aselectable user interface object for performing the operation associatedwith the content; and, in accordance with a determination that the firstuser interface is not configured to perform the operation associatedwith the content, displaying the virtual model of the content withoutdisplaying the selectable user interface object for performing theoperation associated with the content.

In accordance with some embodiments, an electronic device includes adisplay generation component, one or more input devices, optionally oneor more cameras, one or more processors, and memory storing one or moreprograms; the one or more programs are configured to be executed by theone or more processors and the one or more programs include instructionsfor performing or causing performance of the operations of any of themethods described herein. In accordance with some embodiments, acomputer readable storage medium has stored therein instructions, which,when executed by an electronic device with a display generationcomponent, one or more input devices, and optionally one or more camerascause the device to perform or cause performance of the operations ofany of the methods described herein. In accordance with someembodiments, a graphical user interface on an electronic device with adisplay generation component, one or more input devices, optionally oneor more cameras, a memory, and one or more processors to execute one ormore programs stored in the memory includes one or more of the elementsdisplayed in any of the methods described herein, which are updated inresponse to inputs, as described in any of the methods described herein.In accordance with some embodiments, an electronic device includes:display generation component, one or more input devices, and optionallyone or more cameras; and means for performing or causing performance ofthe operations of any of the methods described herein. In accordancewith some embodiments, an information processing apparatus, for use inan electronic device with a display generation component, one or moreinput devices, and optionally one or more cameras includes means forperforming or causing performance of the operations of any of themethods described herein.

Thus, electronic devices with display generation component, one or moreinput devices, and optionally one or more cameras, are provided withimproved methods and interfaces for displaying virtual objects in avariety of contexts, thereby increasing the effectiveness, efficiency,and user satisfaction with such devices. Such methods and interfaces maycomplement or replace conventional methods for displaying virtualobjects in a variety of contexts.

BRIEF DESCRIPTION OF THE DRAWINGS

For a better understanding of the various described embodiments,reference should be made to the Description of Embodiments below, inconjunction with the following drawings in which like reference numeralsrefer to corresponding parts throughout the figures.

FIG. 1A is a block diagram illustrating a portable multifunction devicewith a touch-sensitive display in accordance with some embodiments.

FIG. 1B is a block diagram illustrating example components for eventhandling in accordance with some embodiments.

FIG. 1C is a block diagram illustrating a tactile output module inaccordance with some embodiments.

FIG. 2 illustrates a portable multifunction device having a touch screenin accordance with some embodiments.

FIG. 3 is a block diagram of an example multifunction device with adisplay and a touch-sensitive surface in accordance with someembodiments.

FIG. 4A illustrates an example user interface for a menu of applicationson a portable multifunction device in accordance with some embodiments.

FIG. 4B illustrates an example user interface for a multifunction devicewith a touch-sensitive surface that is separate from the display inaccordance with some embodiments.

FIGS. 4C-4E illustrate examples of dynamic intensity thresholds inaccordance with some embodiments.

FIGS. 4F-4K illustrate a set of sample tactile output patterns inaccordance with some embodiments.

FIGS. 5A-5K, 5L-1, 5L-2, 5M-1, 5M-2, 5N-1, 5N-2, 5O-1, 5O-2, and 5P-5AKillustrate example user interfaces for displaying a visual indication ofone or more inputs that if performed would cause criteria to besatisfied for performing an action, in accordance with some embodiments.

FIGS. 6A-6AI illustrate example user interfaces for displaying a mediaitem in a user interface that includes content of at least a portion ofa field of view of one or more cameras, in accordance with someembodiments.

FIGS. 7A-7AQ illustrate example user interfaces for responding to aninput directed to a respective virtual object of a plurality ofdisplayed virtual objects, in accordance with some embodiments.

FIGS. 8A-8K illustrate example user interfaces for displaying a promptto change a property of a media item that does not meet compatibilitycriteria for display in an augmented reality environment, in accordancewith some embodiments.

FIGS. 9A-9C are flow diagrams of a process for displaying a visualindication of one or more inputs that if performed would cause criteriato be satisfied for performing an action.

FIGS. 10A-10C are flow diagrams of a process for displaying a media itemin a user interface that includes content of at least a portion of afield of view of one or more cameras, in accordance with someembodiments.

FIGS. 11A-11D are flow diagrams of a process for responding to an inputdirected to a respective virtual object of a plurality of displayedvirtual objects, in accordance with some embodiments.

FIGS. 12A-12B are flow diagrams of a process for displaying a prompt tochange a property of a media item that does not meet compatibilitycriteria for display in an augmented reality environment, in accordancewith some embodiments.

FIGS. 13A-13J illustrate example user interfaces for displaying avirtual model of content concurrently with a selectable user interfaceobject for performing an operation associated with the content, inaccordance with some embodiments.

FIGS. 14A-14B are flow diagrams of a process for displaying a virtualmodel of content concurrently with a selectable user interface objectfor performing an operation associated with the content, in accordancewith some embodiments.

DESCRIPTION OF EMBODIMENTS

A virtual object is a graphical representation of a three-dimensionalobject in a virtual environment. Conventional methods of interactingwith virtual objects often lack sufficient feedback to indicate to auser the needed input to achieve an intended outcome (e.g., performingan action associated with a virtual object). Further, conventionalmethods of require a cumbersome series of inputs (input for opening aconversion application, importing a converted item into an applicationfor viewing the converted item in an augmented reality environment,etc.) for adapting media items for viewing in an augmented realityenvironment (e.g., an environment in which a view of the physical worldis augmented with supplemental information that provides additionalinformation to a user that is not available in the physical world). Theembodiments herein provide an intuitive way for a user to displayvirtual objects in various contexts.

The systems, methods, and GUIs described herein improve user interfaceinteractions with virtual/augmented reality environments in multipleways. For example, they make it easier to: display a virtual object inan augmented reality environment and, in response to different inputs,adjust the appearance of the virtual object for display in the augmentedreality environment.

Below, FIGS. 1A-1C, 2, and 3 provide a description of example devices.FIGS. 4A-4B, 5A-5AK, 6A-6AI, 7A-7AQ, 8A-8K, and 13A-13J illustrateexample user interfaces for displaying virtual objects in a variety ofcontexts. FIGS. 9A-9C illustrate flow diagrams of a method of displayinga visual indication of one or more inputs that if performed would causecriteria to be satisfied for performing an action. FIGS. 10A-10Cillustrate flow diagrams of a method of displaying a media item in auser interface that includes content of at least a portion of a field ofview of one or more cameras, in accordance with some embodiments. FIGS.11A-11D illustrate flow diagrams of a method of responding to an inputdirected to a respective virtual object of a plurality of displayedvirtual objects, in accordance with some embodiments. FIGS. 12A-12Billustrate flow diagrams of a method of displaying a prompt to change aproperty of a media item that does not meet compatibility criteria fordisplay in an augmented reality environment, in accordance with someembodiments. The user interfaces in FIGS. 5A-5AK, 6A-6AI, 7A-7AQ, 8A-8K,and 13A-13J are used to illustrate the processes in FIGS. 9A-9C,10A-10C, 11A-11D, 12A-12D and 14A-14B.

Example Devices

Reference will now be made in detail to embodiments, examples of whichare illustrated in the accompanying drawings. In the following detaileddescription, numerous specific details are set forth in order to providea thorough understanding of the various described embodiments. However,it will be apparent to one of ordinary skill in the art that the variousdescribed embodiments may be practiced without these specific details.In other instances, well-known methods, procedures, components,circuits, and networks have not been described in detail so as not tounnecessarily obscure aspects of the embodiments.

It will also be understood that, although the terms first, second, etc.are, in some instances, used herein to describe various elements, theseelements should not be limited by these terms. These terms are only usedto distinguish one element from another. For example, a first contactcould be termed a second contact, and, similarly, a second contact couldbe termed a first contact, without departing from the scope of thevarious described embodiments. The first contact and the second contactare both contacts, but they are not the same contact, unless the contextclearly indicates otherwise.

The terminology used in the description of the various describedembodiments herein is for the purpose of describing particularembodiments only and is not intended to be limiting. As used in thedescription of the various described embodiments and the appendedclaims, the singular forms “a,” “an,” and “the” are intended to includethe plural forms as well, unless the context clearly indicatesotherwise. It will also be understood that the term “and/or” as usedherein refers to and encompasses any and all possible combinations ofone or more of the associated listed items. It will be furtherunderstood that the terms “includes,” “including,” “comprises,” and/or“comprising,” when used in this specification, specify the presence ofstated features, integers, steps, operations, elements, and/orcomponents, but do not preclude the presence or addition of one or moreother features, integers, steps, operations, elements, components,and/or groups thereof.

As used herein, the term “if” is, optionally, construed to mean “when”or “upon” or “in response to determining” or “in response to detecting,”depending on the context. Similarly, the phrase “if it is determined” or“if [a stated condition or event] is detected” is, optionally, construedto mean “upon determining” or “in response to determining” or “upondetecting [the stated condition or event]” or “in response to detecting[the stated condition or event],” depending on the context.

Embodiments of electronic devices, user interfaces for such devices, andassociated processes for using such devices are described. In someembodiments, the device is a portable communications device, such as amobile telephone, that also contains other functions, such as PDA and/ormusic player functions. Example embodiments of portable multifunctiondevices include, without limitation, the iPhone®, iPod Touch®, and iPad®devices from Apple Inc. of Cupertino, Calif. Other portable electronicdevices, such as laptops or tablet computers with touch-sensitivesurfaces (e.g., touch-screen displays and/or touchpads), are,optionally, used. It should also be understood that, in someembodiments, the device is not a portable communications device, but isa desktop computer with a touch-sensitive surface (e.g., a touch-screendisplay and/or a touchpad).

In the discussion that follows, an electronic device that includes adisplay and a touch-sensitive surface is described. It should beunderstood, however, that the electronic device optionally includes oneor more other physical user-interface devices, such as a physicalkeyboard, a mouse and/or a joystick.

The device typically supports a variety of applications, such as one ormore of the following: a note taking application, a drawing application,a presentation application, a word processing application, a websitecreation application, a disk authoring application, a spreadsheetapplication, a gaming application, a telephone application, a videoconferencing application, an e-mail application, an instant messagingapplication, a workout support application, a photo managementapplication, a digital camera application, a digital video cameraapplication, a web browsing application, a digital music playerapplication, and/or a digital video player application.

The various applications that are executed on the device optionally useat least one common physical user-interface device, such as thetouch-sensitive surface. One or more functions of the touch-sensitivesurface as well as corresponding information displayed on the deviceare, optionally, adjusted and/or varied from one application to the nextand/or within a respective application. In this way, a common physicalarchitecture (such as the touch-sensitive surface) of the deviceoptionally supports the variety of applications with user interfacesthat are intuitive and transparent to the user.

Attention is now directed toward embodiments of portable devices withtouch-sensitive displays. FIG. 1A is a block diagram illustratingportable multifunction device 100 with touch-sensitive display system112 in accordance with some embodiments. Touch-sensitive display system112 is sometimes called a “touch screen” for convenience, and issometimes simply called a touch-sensitive display. Device 100 includesmemory 102 (which optionally includes one or more computer readablestorage mediums), memory controller 122, one or more processing units(CPUs) 120, peripherals interface 118, RF circuitry 108, audio circuitry110, speaker 111, microphone 113, input/output (I/O) subsystem 106,other input or control devices 116, and external port 124. Device 100optionally includes one or more optical sensors 164. Device 100optionally includes one or more intensity sensors 165 for detectingintensities of contacts on device 100 (e.g., a touch-sensitive surfacesuch as touch-sensitive display system 112 of device 100). Device 100optionally includes one or more tactile output generators 167 forgenerating tactile outputs on device 100 (e.g., generating tactileoutputs on a touch-sensitive surface such as touch-sensitive displaysystem 112 of device 100 or touchpad 355 of device 300). Thesecomponents optionally communicate over one or more communication busesor signal lines 103.

It should be appreciated that device 100 is only one example of aportable multifunction device, and that device 100 optionally has moreor fewer components than shown, optionally combines two or morecomponents, or optionally has a different configuration or arrangementof the components. The various components shown in FIG. 1A areimplemented in hardware, software, firmware, or a combination thereof,including one or more signal processing and/or application specificintegrated circuits.

Memory 102 optionally includes high-speed random access memory andoptionally also includes non-volatile memory, such as one or moremagnetic disk storage devices, flash memory devices, or othernon-volatile solid-state memory devices. Access to memory 102 by othercomponents of device 100, such as CPU(s) 120 and the peripheralsinterface 118, is, optionally, controlled by memory controller 122.

Peripherals interface 118 can be used to couple input and outputperipherals of the device to CPU(s) 120 and memory 102. The one or moreprocessors 120 run or execute various software programs and/or sets ofinstructions stored in memory 102 to perform various functions fordevice 100 and to process data.

In some embodiments, peripherals interface 118, CPU(s) 120, and memorycontroller 122 are, optionally, implemented on a single chip, such aschip 104. In some other embodiments, they are, optionally, implementedon separate chips.

RF (radio frequency) circuitry 108 receives and sends RF signals, alsocalled electromagnetic signals. RF circuitry 108 converts electricalsignals to/from electromagnetic signals and communicates withcommunications networks and other communications devices via theelectromagnetic signals. RF circuitry 108 optionally includes well-knowncircuitry for performing these functions, including but not limited toan antenna system, an RF transceiver, one or more amplifiers, a tuner,one or more oscillators, a digital signal processor, a CODEC chipset, asubscriber identity module (SIM) card, memory, and so forth. RFcircuitry 108 optionally communicates with networks, such as theInternet, also referred to as the World Wide Web (WWW), an intranetand/or a wireless network, such as a cellular telephone network, awireless local area network (LAN) and/or a metropolitan area network(MAN), and other devices by wireless communication. The wirelesscommunication optionally uses any of a plurality of communicationsstandards, protocols and technologies, including but not limited toGlobal System for Mobile Communications (GSM), Enhanced Data GSMEnvironment (EDGE), high-speed downlink packet access (HSDPA),high-speed uplink packet access (HSDPA), Evolution, Data-Only (EV-DO),HSPA, HSPA+, Dual-Cell HSPA (DC-HSPA), long term evolution (LTE), nearfield communication (NFC), wideband code division multiple access(W-CDMA), code division multiple access (CDMA), time division multipleaccess (TDMA), Bluetooth, Wireless Fidelity (Wi-Fi) (e.g., IEEE 802.11a,IEEE 802.11ac, IEEE 802.11ax, IEEE 802.11b, IEEE 802.11g and/or IEEE802.11n), voice over Internet Protocol (VoIP), Wi-MAX, a protocol fore-mail (e.g., Internet message access protocol (IMAP) and/or post officeprotocol (POP)), instant messaging (e.g., extensible messaging andpresence protocol (XMPP), Session Initiation Protocol for InstantMessaging and Presence Leveraging Extensions (SIMPLE), Instant Messagingand Presence Service (IMPS)), and/or Short Message Service (SMS), or anyother suitable communication protocol, including communication protocolsnot yet developed as of the filing date of this document.

Audio circuitry 110, speaker 111, and microphone 113 provide an audiointerface between a user and device 100. Audio circuitry 110 receivesaudio data from peripherals interface 118, converts the audio data to anelectrical signal, and transmits the electrical signal to speaker 111.Speaker 111 converts the electrical signal to human-audible sound waves.Audio circuitry 110 also receives electrical signals converted bymicrophone 113 from sound waves. Audio circuitry 110 converts theelectrical signal to audio data and transmits the audio data toperipherals interface 118 for processing. Audio data is, optionally,retrieved from and/or transmitted to memory 102 and/or RF circuitry 108by peripherals interface 118. In some embodiments, audio circuitry 110also includes a headset jack (e.g., 212, FIG. 2). The headset jackprovides an interface between audio circuitry 110 and removable audioinput/output peripherals, such as output-only headphones or a headsetwith both output (e.g., a headphone for one or both ears) and input(e.g., a microphone).

I/O subsystem 106 couples input/output peripherals on device 100, suchas touch-sensitive display system 112 and other input or control devices116, with peripherals interface 118. I/O subsystem 106 optionallyincludes display controller 156, optical sensor controller 158,intensity sensor controller 159, haptic feedback controller 161, and oneor more input controllers 160 for other input or control devices. Theone or more input controllers 160 receive/send electrical signalsfrom/to other input or control devices 116. The other input or controldevices 116 optionally include physical buttons (e.g., push buttons,rocker buttons, etc.), dials, slider switches, joysticks, click wheels,and so forth. In some alternate embodiments, input controller(s) 160are, optionally, coupled with any (or none) of the following: akeyboard, infrared port, USB port, stylus, and/or a pointer device suchas a mouse. The one or more buttons (e.g., 208, FIG. 2) optionallyinclude an up/down button for volume control of speaker 111 and/ormicrophone 113. The one or more buttons optionally include a push button(e.g., 206, FIG. 2).

Touch-sensitive display system 112 provides an input interface and anoutput interface between the device and a user. Display controller 156receives and/or sends electrical signals from/to touch-sensitive displaysystem 112. Touch-sensitive display system 112 displays visual output tothe user. The visual output optionally includes graphics, text, icons,video, and any combination thereof (collectively termed “graphics”). Insome embodiments, some or all of the visual output corresponds to userinterface objects. As used herein, the term “affordance” refers to auser-interactive graphical user interface object (e.g., a graphical userinterface object that is configured to respond to inputs directed towardthe graphical user interface object). Examples of user-interactivegraphical user interface objects include, without limitation, a button,slider, icon, selectable menu item, switch, hyperlink, or other userinterface control.

Touch-sensitive display system 112 has a touch-sensitive surface, sensoror set of sensors that accepts input from the user based on hapticand/or tactile contact. Touch-sensitive display system 112 and displaycontroller 156 (along with any associated modules and/or sets ofinstructions in memory 102) detect contact (and any movement or breakingof the contact) on touch-sensitive display system 112 and converts thedetected contact into interaction with user-interface objects (e.g., oneor more soft keys, icons, web pages or images) that are displayed ontouch-sensitive display system 112. In some embodiments, a point ofcontact between touch-sensitive display system 112 and the usercorresponds to a finger of the user or a stylus.

Touch-sensitive display system 112 optionally uses LCD (liquid crystaldisplay) technology, LPD (light emitting polymer display) technology, orLED (light emitting diode) technology, although other displaytechnologies are used in other embodiments. Touch-sensitive displaysystem 112 and display controller 156 optionally detect contact and anymovement or breaking thereof using any of a plurality of touch sensingtechnologies now known or later developed, including but not limited tocapacitive, resistive, infrared, and surface acoustic wave technologies,as well as other proximity sensor arrays or other elements fordetermining one or more points of contact with touch-sensitive displaysystem 112. In some embodiments, projected mutual capacitance sensingtechnology is used, such as that found in the iPhone®, iPod Touch®, andiPad® from Apple Inc. of Cupertino, Calif.

Touch-sensitive display system 112 optionally has a video resolution inexcess of 100 dpi. In some embodiments, the touch screen videoresolution is in excess of 400 dpi (e.g., 500 dpi, 800 dpi, or greater).The user optionally makes contact with touch-sensitive display system112 using any suitable object or appendage, such as a stylus, a finger,and so forth. In some embodiments, the user interface is designed towork with finger-based contacts and gestures, which can be less precisethan stylus-based input due to the larger area of contact of a finger onthe touch screen. In some embodiments, the device translates the roughfinger-based input into a precise pointer/cursor position or command forperforming the actions desired by the user.

In some embodiments, in addition to the touch screen, device 100optionally includes a touchpad (not shown) for activating ordeactivating particular functions. In some embodiments, the touchpad isa touch-sensitive area of the device that, unlike the touch screen, doesnot display visual output. The touchpad is, optionally, atouch-sensitive surface that is separate from touch-sensitive displaysystem 112 or an extension of the touch-sensitive surface formed by thetouch screen.

Device 100 also includes power system 162 for powering the variouscomponents. Power system 162 optionally includes a power managementsystem, one or more power sources (e.g., battery, alternating current(AC)), a recharging system, a power failure detection circuit, a powerconverter or inverter, a power status indicator (e.g., a light-emittingdiode (LED)) and any other components associated with the generation,management and distribution of power in portable devices.

Device 100 optionally also includes one or more optical sensors 164.FIG. 1A shows an optical sensor coupled with optical sensor controller158 in I/O subsystem 106. Optical sensor(s) 164 optionally includecharge-coupled device (CCD) or complementary metal-oxide semiconductor(CMOS) phototransistors. Optical sensor(s) 164 receive light from theenvironment, projected through one or more lens, and converts the lightto data representing an image. In conjunction with imaging module 143(also called a camera module), optical sensor(s) 164 optionally capturestill images and/or video. In some embodiments, an optical sensor islocated on the back of device 100, opposite touch-sensitive displaysystem 112 on the front of the device, so that the touch screen isenabled for use as a viewfinder for still and/or video imageacquisition. In some embodiments, another optical sensor is located onthe front of the device so that the user's image is obtained (e.g., forselfies, for videoconferencing while the user views the other videoconference participants on the touch screen, etc.).

Device 100 optionally also includes one or more contact intensitysensors 165. FIG. 1A shows a contact intensity sensor coupled withintensity sensor controller 159 in I/O subsystem 106. Contact intensitysensor(s) 165 optionally include one or more piezoresistive straingauges, capacitive force sensors, electric force sensors, piezoelectricforce sensors, optical force sensors, capacitive touch-sensitivesurfaces, or other intensity sensors (e.g., sensors used to measure theforce (or pressure) of a contact on a touch-sensitive surface). Contactintensity sensor(s) 165 receive contact intensity information (e.g.,pressure information or a proxy for pressure information) from theenvironment. In some embodiments, at least one contact intensity sensoris collocated with, or proximate to, a touch-sensitive surface (e.g.,touch-sensitive display system 112). In some embodiments, at least onecontact intensity sensor is located on the back of device 100, oppositetouch-screen display system 112 which is located on the front of device100.

Device 100 optionally also includes one or more proximity sensors 166.FIG. 1A shows proximity sensor 166 coupled with peripherals interface118. Alternately, proximity sensor 166 is coupled with input controller160 in I/O subsystem 106. In some embodiments, the proximity sensorturns off and disables touch-sensitive display system 112 when themultifunction device is placed near the user's ear (e.g., when the useris making a phone call).

Device 100 optionally also includes one or more tactile outputgenerators 167. FIG. 1A shows a tactile output generator coupled withhaptic feedback controller 161 in I/O subsystem 106. In someembodiments, tactile output generator(s) 167 include one or moreelectroacoustic devices such as speakers or other audio componentsand/or electromechanical devices that convert energy into linear motionsuch as a motor, solenoid, electroactive polymer, piezoelectricactuator, electrostatic actuator, or other tactile output generatingcomponent (e.g., a component that converts electrical signals intotactile outputs on the device). Tactile output generator(s) 167 receivetactile feedback generation instructions from haptic feedback module 133and generates tactile outputs on device 100 that are capable of beingsensed by a user of device 100. In some embodiments, at least onetactile output generator is collocated with, or proximate to, atouch-sensitive surface (e.g., touch-sensitive display system 112) and,optionally, generates a tactile output by moving the touch-sensitivesurface vertically (e.g., in/out of a surface of device 100) orlaterally (e.g., back and forth in the same plane as a surface of device100). In some embodiments, at least one tactile output generator sensoris located on the back of device 100, opposite touch-sensitive displaysystem 112, which is located on the front of device 100.

Device 100 optionally also includes one or more accelerometers 168. FIG.1A shows accelerometer 168 coupled with peripherals interface 118.Alternately, accelerometer 168 is, optionally, coupled with an inputcontroller 160 in I/O subsystem 106. In some embodiments, information isdisplayed on the touch-screen display in a portrait view or a landscapeview based on an analysis of data received from the one or moreaccelerometers. Device 100 optionally includes, in addition toaccelerometer(s) 168, a magnetometer (not shown) and a GPS (or GLONASSor other global navigation system) receiver (not shown) for obtaininginformation concerning the location and orientation (e.g., portrait orlandscape) of device 100.

In some embodiments, the software components stored in memory 102include operating system 126, communication module (or set ofinstructions) 128, contact/motion module (or set of instructions) 130,graphics module (or set of instructions) 132, haptic feedback module (orset of instructions) 133, text input module (or set of instructions)134, Global Positioning System (GPS) module (or set of instructions)135, and applications (or sets of instructions) 136. Furthermore, insome embodiments, memory 102 stores device/global internal state 157, asshown in FIGS. 1A and 3. Device/global internal state 157 includes oneor more of: active application state, indicating which applications, ifany, are currently active; display state, indicating what applications,views or other information occupy various regions of touch-sensitivedisplay system 112; sensor state, including information obtained fromthe device's various sensors and other input or control devices 116; andlocation and/or positional information concerning the device's locationand/or attitude.

Operating system 126 (e.g., iOS, Darwin, RTXC, LINUX, UNIX, OS X,WINDOWS, or an embedded operating system such as VxWorks) includesvarious software components and/or drivers for controlling and managinggeneral system tasks (e.g., memory management, storage device control,power management, etc.) and facilitates communication between varioushardware and software components.

Communication module 128 facilitates communication with other devicesover one or more external ports 124 and also includes various softwarecomponents for handling data received by RF circuitry 108 and/orexternal port 124. External port 124 (e.g., Universal Serial Bus (USB),FIREWIRE, etc.) is adapted for coupling directly to other devices orindirectly over a network (e.g., the Internet, wireless LAN, etc.). Insome embodiments, the external port is a multi-pin (e.g., 30-pin)connector that is the same as, or similar to and/or compatible with the30-pin connector used in some iPhone®, iPod Touch®, and iPad® devicesfrom Apple Inc. of Cupertino, Calif. In some embodiments, the externalport is a Lightning connector that is the same as, or similar to and/orcompatible with the Lightning connector used in some iPhone®, iPodTouch®, and iPad® devices from Apple Inc. of Cupertino, Calif.

Contact/motion module 130 optionally detects contact withtouch-sensitive display system 112 (in conjunction with displaycontroller 156) and other touch-sensitive devices (e.g., a touchpad orphysical click wheel). Contact/motion module 130 includes varioussoftware components for performing various operations related todetection of contact (e.g., by a finger or by a stylus), such asdetermining if contact has occurred (e.g., detecting a finger-downevent), determining an intensity of the contact (e.g., the force orpressure of the contact or a substitute for the force or pressure of thecontact), determining if there is movement of the contact and trackingthe movement across the touch-sensitive surface (e.g., detecting one ormore finger-dragging events), and determining if the contact has ceased(e.g., detecting a finger-up event or a break in contact).Contact/motion module 130 receives contact data from the touch-sensitivesurface. Determining movement of the point of contact, which isrepresented by a series of contact data, optionally includes determiningspeed (magnitude), velocity (magnitude and direction), and/or anacceleration (a change in magnitude and/or direction) of the point ofcontact. These operations are, optionally, applied to single contacts(e.g., one finger contacts or stylus contacts) or to multiplesimultaneous contacts (e.g., “multitouch”/multiple finger contacts). Insome embodiments, contact/motion module 130 and display controller 156detect contact on a touchpad.

Contact/motion module 130 optionally detects a gesture input by a user.Different gestures on the touch-sensitive surface have different contactpatterns (e.g., different motions, timings, and/or intensities ofdetected contacts). Thus, a gesture is, optionally, detected bydetecting a particular contact pattern. For example, detecting a fingertap gesture includes detecting a finger-down event followed by detectinga finger-up (lift off) event at the same position (or substantially thesame position) as the finger-down event (e.g., at the position of anicon). As another example, detecting a finger swipe gesture on thetouch-sensitive surface includes detecting a finger-down event followedby detecting one or more finger-dragging events, and subsequentlyfollowed by detecting a finger-up (lift off) event. Similarly, tap,swipe, drag, and other gestures are optionally detected for a stylus bydetecting a particular contact pattern for the stylus.

In some embodiments, detecting a finger tap gesture depends on thelength of time between detecting the finger-down event and the finger-upevent, but is independent of the intensity of the finger contact betweendetecting the finger-down event and the finger-up event. In someembodiments, a tap gesture is detected in accordance with adetermination that the length of time between the finger-down event andthe finger-up event is less than a predetermined value (e.g., less than0.1, 0.2, 0.3, 0.4 or 0.5 seconds), independent of whether the intensityof the finger contact during the tap meets a given intensity threshold(greater than a nominal contact-detection intensity threshold), such asa light press or deep press intensity threshold. Thus, a finger tapgesture can satisfy particular input criteria that do not require thatthe characteristic intensity of a contact satisfy a given intensitythreshold in order for the particular input criteria to be met. Forclarity, the finger contact in a tap gesture typically needs to satisfya nominal contact-detection intensity threshold, below which the contactis not detected, in order for the finger-down event to be detected. Asimilar analysis applies to detecting a tap gesture by a stylus or othercontact. In cases where the device is capable of detecting a finger orstylus contact hovering over a touch sensitive surface, the nominalcontact-detection intensity threshold optionally does not correspond tophysical contact between the finger or stylus and the touch sensitivesurface.

The same concepts apply in an analogous manner to other types ofgestures. For example, a swipe gesture, a pinch gesture, a depinchgesture, and/or a long press gesture are optionally detected based onthe satisfaction of criteria that are either independent of intensitiesof contacts included in the gesture, or do not require that contact(s)that perform the gesture reach intensity thresholds in order to berecognized. For example, a swipe gesture is detected based on an amountof movement of one or more contacts; a pinch gesture is detected basedon movement of two or more contacts towards each other; a depinchgesture is detected based on movement of two or more contacts away fromeach other; and a long press gesture is detected based on a duration ofthe contact on the touch-sensitive surface with less than a thresholdamount of movement. As such, the statement that particular gesturerecognition criteria do not require that the intensity of the contact(s)meet a respective intensity threshold in order for the particulargesture recognition criteria to be met means that the particular gesturerecognition criteria are capable of being satisfied if the contact(s) inthe gesture do not reach the respective intensity threshold, and arealso capable of being satisfied in circumstances where one or more ofthe contacts in the gesture do reach or exceed the respective intensitythreshold. In some embodiments, a tap gesture is detected based on adetermination that the finger-down and finger-up event are detectedwithin a predefined time period, without regard to whether the contactis above or below the respective intensity threshold during thepredefined time period, and a swipe gesture is detected based on adetermination that the contact movement is greater than a predefinedmagnitude, even if the contact is above the respective intensitythreshold at the end of the contact movement. Even in implementationswhere detection of a gesture is influenced by the intensity of contactsperforming the gesture (e.g., the device detects a long press morequickly when the intensity of the contact is above an intensitythreshold or delays detection of a tap input when the intensity of thecontact is higher), the detection of those gestures does not requirethat the contacts reach a particular intensity threshold so long as thecriteria for recognizing the gesture can be met in circumstances wherethe contact does not reach the particular intensity threshold (e.g.,even if the amount of time that it takes to recognize the gesturechanges).

Contact intensity thresholds, duration thresholds, and movementthresholds are, in some circumstances, combined in a variety ofdifferent combinations in order to create heuristics for distinguishingtwo or more different gestures directed to the same input element orregion so that multiple different interactions with the same inputelement are enabled to provide a richer set of user interactions andresponses. The statement that a particular set of gesture recognitioncriteria do not require that the intensity of the contact(s) meet arespective intensity threshold in order for the particular gesturerecognition criteria to be met does not preclude the concurrentevaluation of other intensity-dependent gesture recognition criteria toidentify other gestures that do have criteria that are met when agesture includes a contact with an intensity above the respectiveintensity threshold. For example, in some circumstances, first gesturerecognition criteria for a first gesture—which do not require that theintensity of the contact(s) meet a respective intensity threshold inorder for the first gesture recognition criteria to be met—are incompetition with second gesture recognition criteria for a secondgesture—which are dependent on the contact(s) reaching the respectiveintensity threshold. In such competitions, the gesture is, optionally,not recognized as meeting the first gesture recognition criteria for thefirst gesture if the second gesture recognition criteria for the secondgesture are met first. For example, if a contact reaches the respectiveintensity threshold before the contact moves by a predefined amount ofmovement, a deep press gesture is detected rather than a swipe gesture.Conversely, if the contact moves by the predefined amount of movementbefore the contact reaches the respective intensity threshold, a swipegesture is detected rather than a deep press gesture. Even in suchcircumstances, the first gesture recognition criteria for the firstgesture still do not require that the intensity of the contact(s) meet arespective intensity threshold in order for the first gesturerecognition criteria to be met because if the contact stayed below therespective intensity threshold until an end of the gesture (e.g., aswipe gesture with a contact that does not increase to an intensityabove the respective intensity threshold), the gesture would have beenrecognized by the first gesture recognition criteria as a swipe gesture.As such, particular gesture recognition criteria that do not requirethat the intensity of the contact(s) meet a respective intensitythreshold in order for the particular gesture recognition criteria to bemet will (A) in some circumstances ignore the intensity of the contactwith respect to the intensity threshold (e.g. for a tap gesture) and/or(B) in some circumstances still be dependent on the intensity of thecontact with respect to the intensity threshold in the sense that theparticular gesture recognition criteria (e.g., for a long press gesture)will fail if a competing set of intensity-dependent gesture recognitioncriteria (e.g., for a deep press gesture) recognize an input ascorresponding to an intensity-dependent gesture before the particulargesture recognition criteria recognize a gesture corresponding to theinput (e.g., for a long press gesture that is competing with a deeppress gesture for recognition).

Graphics module 132 includes various known software components forrendering and displaying graphics on touch-sensitive display system 112or other display, including components for changing the visual impact(e.g., brightness, transparency, saturation, contrast or other visualproperty) of graphics that are displayed. As used herein, the term“graphics” includes any object that can be displayed to a user,including without limitation text, web pages, icons (such asuser-interface objects including soft keys), digital images, videos,animations and the like.

In some embodiments, graphics module 132 stores data representinggraphics to be used. Each graphic is, optionally, assigned acorresponding code. Graphics module 132 receives, from applicationsetc., one or more codes specifying graphics to be displayed along with,if necessary, coordinate data and other graphic property data, and thengenerates screen image data to output to display controller 156.

Haptic feedback module 133 includes various software components forgenerating instructions (e.g., instructions used by haptic feedbackcontroller 161) to produce tactile outputs using tactile outputgenerator(s) 167 at one or more locations on device 100 in response touser interactions with device 100.

Text input module 134, which is, optionally, a component of graphicsmodule 132, provides soft keyboards for entering text in variousapplications (e.g., contacts 137, e-mail 140, IM 141, browser 147, andany other application that needs text input).

GPS module 135 determines the location of the device and provides thisinformation for use in various applications (e.g., to telephone 138 foruse in location-based dialing, to camera 143 as picture/video metadata,and to applications that provide location-based services such as weatherwidgets, local yellow page widgets, and map/navigation widgets).

Applications 136 optionally include the following modules (or sets ofinstructions), or a subset or superset thereof:

-   -   contacts module 137 (sometimes called an address book or contact        list);    -   telephone module 138;    -   video conferencing module 139;    -   e-mail client module 140;    -   instant messaging (IM) module 141;    -   workout support module 142;    -   camera module 143 for still and/or video images;    -   image management module 144;    -   browser module 147;    -   calendar module 148;    -   widget modules 149, which optionally include one or more of:        weather widget 149-1, stocks widget 149-2, calculator widget        149-3, alarm clock widget 149-4, dictionary widget 149-5, and        other widgets obtained by the user, as well as user-created        widgets 149-6;    -   widget creator module 150 for making user-created widgets 149-6;    -   search module 151;    -   video and music player module 152, which is, optionally, made up        of a video player module and a music player module;    -   notes module 153;    -   map module 154; and/or    -   online video module 155.

Examples of other applications 136 that are, optionally, stored inmemory 102 include other word processing applications, other imageediting applications, drawing applications, presentation applications,JAVA-enabled applications, encryption, digital rights management, voicerecognition, and voice replication.

In conjunction with touch-sensitive display system 112, displaycontroller 156, contact module 130, graphics module 132, and text inputmodule 134, contacts module 137 includes executable instructions tomanage an address book or contact list (e.g., stored in applicationinternal state 192 of contacts module 137 in memory 102 or memory 370),including: adding name(s) to the address book; deleting name(s) from theaddress book; associating telephone number(s), e-mail address(es),physical address(es) or other information with a name; associating animage with a name; categorizing and sorting names; providing telephonenumbers and/or e-mail addresses to initiate and/or facilitatecommunications by telephone 138, video conference 139, e-mail 140, or IM141; and so forth.

In conjunction with RF circuitry 108, audio circuitry 110, speaker 111,microphone 113, touch-sensitive display system 112, display controller156, contact module 130, graphics module 132, and text input module 134,telephone module 138 includes executable instructions to enter asequence of characters corresponding to a telephone number, access oneor more telephone numbers in address book 137, modify a telephone numberthat has been entered, dial a respective telephone number, conduct aconversation and disconnect or hang up when the conversation iscompleted. As noted above, the wireless communication optionally usesany of a plurality of communications standards, protocols andtechnologies.

In conjunction with RF circuitry 108, audio circuitry 110, speaker 111,microphone 113, touch-sensitive display system 112, display controller156, optical sensor(s) 164, optical sensor controller 158, contactmodule 130, graphics module 132, text input module 134, contact list137, and telephone module 138, videoconferencing module 139 includesexecutable instructions to initiate, conduct, and terminate a videoconference between a user and one or more other participants inaccordance with user instructions.

In conjunction with RF circuitry 108, touch-sensitive display system112, display controller 156, contact module 130, graphics module 132,and text input module 134, e-mail client module 140 includes executableinstructions to create, send, receive, and manage e-mail in response touser instructions. In conjunction with image management module 144,e-mail client module 140 makes it very easy to create and send e-mailswith still or video images taken with camera module 143.

In conjunction with RF circuitry 108, touch-sensitive display system112, display controller 156, contact module 130, graphics module 132,and text input module 134, the instant messaging module 141 includesexecutable instructions to enter a sequence of characters correspondingto an instant message, to modify previously entered characters, totransmit a respective instant message (for example, using a ShortMessage Service (SMS) or Multimedia Message Service (MMS) protocol fortelephony-based instant messages or using XMPP, SIMPLE, Apple PushNotification Service (APNs) or IMPS for Internet-based instantmessages), to receive instant messages, and to view received instantmessages. In some embodiments, transmitted and/or received instantmessages optionally include graphics, photos, audio files, video filesand/or other attachments as are supported in a MMS and/or an EnhancedMessaging Service (EMS). As used herein, “instant messaging” refers toboth telephony-based messages (e.g., messages sent using SMS or MMS) andInternet-based messages (e.g., messages sent using XMPP, SIMPLE, APNs,or IMPS).

In conjunction with RF circuitry 108, touch-sensitive display system112, display controller 156, contact module 130, graphics module 132,text input module 134, GPS module 135, map module 154, and video andmusic player module 152, workout support module 142 includes executableinstructions to create workouts (e.g., with time, distance, and/orcalorie burning goals); communicate with workout sensors (in sportsdevices and smart watches); receive workout sensor data; calibratesensors used to monitor a workout; select and play music for a workout;and display, store and transmit workout data.

In conjunction with touch-sensitive display system 112, displaycontroller 156, optical sensor(s) 164, optical sensor controller 158,contact module 130, graphics module 132, and image management module144, camera module 143 includes executable instructions to capture stillimages or video (including a video stream) and store them into memory102, modify characteristics of a still image or video, and/or delete astill image or video from memory 102.

In conjunction with touch-sensitive display system 112, displaycontroller 156, contact module 130, graphics module 132, text inputmodule 134, and camera module 143, image management module 144 includesexecutable instructions to arrange, modify (e.g., edit), or otherwisemanipulate, label, delete, present (e.g., in a digital slide show oralbum), and store still and/or video images.

In conjunction with RF circuitry 108, touch-sensitive display system112, display system controller 156, contact module 130, graphics module132, and text input module 134, browser module 147 includes executableinstructions to browse the Internet in accordance with userinstructions, including searching, linking to, receiving, and displayingweb pages or portions thereof, as well as attachments and other fileslinked to web pages.

In conjunction with RF circuitry 108, touch-sensitive display system112, display system controller 156, contact module 130, graphics module132, text input module 134, e-mail client module 140, and browser module147, calendar module 148 includes executable instructions to create,display, modify, and store calendars and data associated with calendars(e.g., calendar entries, to do lists, etc.) in accordance with userinstructions.

In conjunction with RF circuitry 108, touch-sensitive display system112, display system controller 156, contact module 130, graphics module132, text input module 134, and browser module 147, widget modules 149are mini-applications that are, optionally, downloaded and used by auser (e.g., weather widget 149-1, stocks widget 149-2, calculator widget149-3, alarm clock widget 149-4, and dictionary widget 149-5) or createdby the user (e.g., user-created widget 149-6). In some embodiments, awidget includes an HTML (Hypertext Markup Language) file, a CSS(Cascading Style Sheets) file, and a JavaScript file. In someembodiments, a widget includes an XML (Extensible Markup Language) fileand a JavaScript file (e.g., Yahoo! Widgets).

In conjunction with RF circuitry 108, touch-sensitive display system112, display system controller 156, contact module 130, graphics module132, text input module 134, and browser module 147, the widget creatormodule 150 includes executable instructions to create widgets (e.g.,turning a user-specified portion of a web page into a widget).

In conjunction with touch-sensitive display system 112, display systemcontroller 156, contact module 130, graphics module 132, and text inputmodule 134, search module 151 includes executable instructions to searchfor text, music, sound, image, video, and/or other files in memory 102that match one or more search criteria (e.g., one or more user-specifiedsearch terms) in accordance with user instructions.

In conjunction with touch-sensitive display system 112, display systemcontroller 156, contact module 130, graphics module 132, audio circuitry110, speaker 111, RF circuitry 108, and browser module 147, video andmusic player module 152 includes executable instructions that allow theuser to download and play back recorded music and other sound filesstored in one or more file formats, such as MP3 or AAC files, andexecutable instructions to display, present or otherwise play backvideos (e.g., on touch-sensitive display system 112, or on an externaldisplay connected wirelessly or via external port 124). In someembodiments, device 100 optionally includes the functionality of an MP3player, such as an iPod (trademark of Apple Inc.).

In conjunction with touch-sensitive display system 112, displaycontroller 156, contact module 130, graphics module 132, and text inputmodule 134, notes module 153 includes executable instructions to createand manage notes, to do lists, and the like in accordance with userinstructions.

In conjunction with RF circuitry 108, touch-sensitive display system112, display system controller 156, contact module 130, graphics module132, text input module 134, GPS module 135, and browser module 147, mapmodule 154 includes executable instructions to receive, display, modify,and store maps and data associated with maps (e.g., driving directions;data on stores and other points of interest at or near a particularlocation; and other location-based data) in accordance with userinstructions.

In conjunction with touch-sensitive display system 112, display systemcontroller 156, contact module 130, graphics module 132, audio circuitry110, speaker 111, RF circuitry 108, text input module 134, e-mail clientmodule 140, and browser module 147, online video module 155 includesexecutable instructions that allow the user to access, browse, receive(e.g., by streaming and/or download), play back (e.g., on thetouch-sensitive display system 112, or on an external display connectedwirelessly or via external port 124), send an e-mail with a link to aparticular online video, and otherwise manage online videos in one ormore file formats, such as H.264. In some embodiments, instant messagingmodule 141, rather than e-mail client module 140, is used to send a linkto a particular online video.

Each of the above identified modules and applications correspond to aset of executable instructions for performing one or more functionsdescribed above and the methods described in this application (e.g., thecomputer-implemented methods and other information processing methodsdescribed herein). These modules (i.e., sets of instructions) need notbe implemented as separate software programs, procedures or modules, andthus various subsets of these modules are, optionally, combined orotherwise re-arranged in various embodiments. In some embodiments,memory 102 optionally stores a subset of the modules and data structuresidentified above. Furthermore, memory 102 optionally stores additionalmodules and data structures not described above.

In some embodiments, device 100 is a device where operation of apredefined set of functions on the device is performed exclusivelythrough a touch screen and/or a touchpad. By using a touch screen and/ora touchpad as the primary input control device for operation of device100, the number of physical input control devices (such as push buttons,dials, and the like) on device 100 is, optionally, reduced.

The predefined set of functions that are performed exclusively through atouch screen and/or a touchpad optionally include navigation betweenuser interfaces. In some embodiments, the touchpad, when touched by theuser, navigates device 100 to a main, home, or root menu from any userinterface that is displayed on device 100. In such embodiments, a “menubutton” is implemented using a touchpad. In some other embodiments, themenu button is a physical push button or other physical input controldevice instead of a touchpad.

FIG. 1B is a block diagram illustrating example components for eventhandling in accordance with some embodiments. In some embodiments,memory 102 (in FIG. 1A) or 370 (FIG. 3) includes event sorter 170 (e.g.,in operating system 126) and a respective application 136-1 (e.g., anyof the aforementioned applications 136, 137-155, 380-390).

Event sorter 170 receives event information and determines theapplication 136-1 and application view 191 of application 136-1 to whichto deliver the event information. Event sorter 170 includes eventmonitor 171 and event dispatcher module 174. In some embodiments,application 136-1 includes application internal state 192, whichindicates the current application view(s) displayed on touch-sensitivedisplay system 112 when the application is active or executing. In someembodiments, device/global internal state 157 is used by event sorter170 to determine which application(s) is (are) currently active, andapplication internal state 192 is used by event sorter 170 to determineapplication views 191 to which to deliver event information.

In some embodiments, application internal state 192 includes additionalinformation, such as one or more of: resume information to be used whenapplication 136-1 resumes execution, user interface state informationthat indicates information being displayed or that is ready for displayby application 136-1, a state queue for enabling the user to go back toa prior state or view of application 136-1, and a redo/undo queue ofprevious actions taken by the user.

Event monitor 171 receives event information from peripherals interface118. Event information includes information about a sub-event (e.g., auser touch on touch-sensitive display system 112, as part of amulti-touch gesture). Peripherals interface 118 transmits information itreceives from I/O subsystem 106 or a sensor, such as proximity sensor166, accelerometer(s) 168, and/or microphone 113 (through audiocircuitry 110). Information that peripherals interface 118 receives fromI/O subsystem 106 includes information from touch-sensitive displaysystem 112 or a touch-sensitive surface.

In some embodiments, event monitor 171 sends requests to the peripheralsinterface 118 at predetermined intervals. In response, peripheralsinterface 118 transmits event information. In other embodiments,peripheral interface 118 transmits event information only when there isa significant event (e.g., receiving an input above a predeterminednoise threshold and/or for more than a predetermined duration).

In some embodiments, event sorter 170 also includes a hit viewdetermination module 172 and/or an active event recognizer determinationmodule 173.

Hit view determination module 172 provides software procedures fordetermining where a sub-event has taken place within one or more views,when touch-sensitive display system 112 displays more than one view.Views are made up of controls and other elements that a user can see onthe display.

Another aspect of the user interface associated with an application is aset of views, sometimes herein called application views or userinterface windows, in which information is displayed and touch-basedgestures occur. The application views (of a respective application) inwhich a touch is detected optionally correspond to programmatic levelswithin a programmatic or view hierarchy of the application. For example,the lowest level view in which a touch is detected is, optionally,called the hit view, and the set of events that are recognized as properinputs are, optionally, determined based, at least in part, on the hitview of the initial touch that begins a touch-based gesture.

Hit view determination module 172 receives information related tosub-events of a touch-based gesture. When an application has multipleviews organized in a hierarchy, hit view determination module 172identifies a hit view as the lowest view in the hierarchy which shouldhandle the sub-event. In most circumstances, the hit view is the lowestlevel view in which an initiating sub-event occurs (i.e., the firstsub-event in the sequence of sub-events that form an event or potentialevent). Once the hit view is identified by the hit view determinationmodule, the hit view typically receives all sub-events related to thesame touch or input source for which it was identified as the hit view.

Active event recognizer determination module 173 determines which viewor views within a view hierarchy should receive a particular sequence ofsub-events. In some embodiments, active event recognizer determinationmodule 173 determines that only the hit view should receive a particularsequence of sub-events. In other embodiments, active event recognizerdetermination module 173 determines that all views that include thephysical location of a sub-event are actively involved views, andtherefore determines that all actively involved views should receive aparticular sequence of sub-events. In other embodiments, even if touchsub-events were entirely confined to the area associated with oneparticular view, views higher in the hierarchy would still remain asactively involved views.

Event dispatcher module 174 dispatches the event information to an eventrecognizer (e.g., event recognizer 180). In embodiments including activeevent recognizer determination module 173, event dispatcher module 174delivers the event information to an event recognizer determined byactive event recognizer determination module 173. In some embodiments,event dispatcher module 174 stores in an event queue the eventinformation, which is retrieved by a respective event receiver module182.

In some embodiments, operating system 126 includes event sorter 170.Alternatively, application 136-1 includes event sorter 170. In yet otherembodiments, event sorter 170 is a stand-alone module, or a part ofanother module stored in memory 102, such as contact/motion module 130.

In some embodiments, application 136-1 includes a plurality of eventhandlers 190 and one or more application views 191, each of whichincludes instructions for handling touch events that occur within arespective view of the application's user interface. Each applicationview 191 of the application 136-1 includes one or more event recognizers180. Typically, a respective application view 191 includes a pluralityof event recognizers 180. In other embodiments, one or more of eventrecognizers 180 are part of a separate module, such as a user interfacekit (not shown) or a higher level object from which application 136-1inherits methods and other properties. In some embodiments, a respectiveevent handler 190 includes one or more of: data updater 176, objectupdater 177, GUI updater 178, and/or event data 179 received from eventsorter 170. Event handler 190 optionally utilizes or calls data updater176, object updater 177 or GUI updater 178 to update the applicationinternal state 192. Alternatively, one or more of the application views191 includes one or more respective event handlers 190. Also, in someembodiments, one or more of data updater 176, object updater 177, andGUI updater 178 are included in a respective application view 191.

A respective event recognizer 180 receives event information (e.g.,event data 179) from event sorter 170, and identifies an event from theevent information. Event recognizer 180 includes event receiver 182 andevent comparator 184. In some embodiments, event recognizer 180 alsoincludes at least a subset of: metadata 183, and event deliveryinstructions 188 (which optionally include sub-event deliveryinstructions).

Event receiver 182 receives event information from event sorter 170. Theevent information includes information about a sub-event, for example, atouch or a touch movement. Depending on the sub-event, the eventinformation also includes additional information, such as location ofthe sub-event. When the sub-event concerns motion of a touch, the eventinformation optionally also includes speed and direction of thesub-event. In some embodiments, events include rotation of the devicefrom one orientation to another (e.g., from a portrait orientation to alandscape orientation, or vice versa), and the event informationincludes corresponding information about the current orientation (alsocalled device attitude) of the device.

Event comparator 184 compares the event information to predefined eventor sub-event definitions and, based on the comparison, determines anevent or sub-event, or determines or updates the state of an event orsub-event. In some embodiments, event comparator 184 includes eventdefinitions 186. Event definitions 186 contain definitions of events(e.g., predefined sequences of sub-events), for example, event 1(187-1), event 2 (187-2), and others. In some embodiments, sub-events inan event 187 include, for example, touch begin, touch end, touchmovement, touch cancellation, and multiple touching. In one example, thedefinition for event 1 (187-1) is a double tap on a displayed object.The double tap, for example, comprises a first touch (touch begin) onthe displayed object for a predetermined phase, a first lift-off (touchend) for a predetermined phase, a second touch (touch begin) on thedisplayed object for a predetermined phase, and a second lift-off (touchend) for a predetermined phase. In another example, the definition forevent 2 (187-2) is a dragging on a displayed object. The dragging, forexample, comprises a touch (or contact) on the displayed object for apredetermined phase, a movement of the touch across touch-sensitivedisplay system 112, and lift-off of the touch (touch end). In someembodiments, the event also includes information for one or moreassociated event handlers 190.

In some embodiments, event definition 187 includes a definition of anevent for a respective user-interface object. In some embodiments, eventcomparator 184 performs a hit test to determine which user-interfaceobject is associated with a sub-event. For example, in an applicationview in which three user-interface objects are displayed ontouch-sensitive display system 112, when a touch is detected ontouch-sensitive display system 112, event comparator 184 performs a hittest to determine which of the three user-interface objects isassociated with the touch (sub-event). If each displayed object isassociated with a respective event handler 190, the event comparatoruses the result of the hit test to determine which event handler 190should be activated. For example, event comparator 184 selects an eventhandler associated with the sub-event and the object triggering the hittest.

In some embodiments, the definition for a respective event 187 alsoincludes delayed actions that delay delivery of the event informationuntil after it has been determined whether the sequence of sub-eventsdoes or does not correspond to the event recognizer's event type.

When a respective event recognizer 180 determines that the series ofsub-events do not match any of the events in event definitions 186, therespective event recognizer 180 enters an event impossible, eventfailed, or event ended state, after which it disregards subsequentsub-events of the touch-based gesture. In this situation, other eventrecognizers, if any, that remain active for the hit view continue totrack and process sub-events of an ongoing touch-based gesture.

In some embodiments, a respective event recognizer 180 includes metadata183 with configurable properties, flags, and/or lists that indicate howthe event delivery system should perform sub-event delivery to activelyinvolved event recognizers. In some embodiments, metadata 183 includesconfigurable properties, flags, and/or lists that indicate how eventrecognizers interact, or are enabled to interact, with one another. Insome embodiments, metadata 183 includes configurable properties, flags,and/or lists that indicate whether sub-events are delivered to varyinglevels in the view or programmatic hierarchy.

In some embodiments, a respective event recognizer 180 activates eventhandler 190 associated with an event when one or more particularsub-events of an event are recognized. In some embodiments, a respectiveevent recognizer 180 delivers event information associated with theevent to event handler 190. Activating an event handler 190 is distinctfrom sending (and deferred sending) sub-events to a respective hit view.In some embodiments, event recognizer 180 throws a flag associated withthe recognized event, and event handler 190 associated with the flagcatches the flag and performs a predefined process.

In some embodiments, event delivery instructions 188 include sub-eventdelivery instructions that deliver event information about a sub-eventwithout activating an event handler. Instead, the sub-event deliveryinstructions deliver event information to event handlers associated withthe series of sub-events or to actively involved views. Event handlersassociated with the series of sub-events or with actively involved viewsreceive the event information and perform a predetermined process.

In some embodiments, data updater 176 creates and updates data used inapplication 136-1. For example, data updater 176 updates the telephonenumber used in contacts module 137, or stores a video file used in videoand music player module 152. In some embodiments, object updater 177creates and updates objects used in application 136-1. For example,object updater 177 creates a new user-interface object or updates theposition of a user-interface object. GUI updater 178 updates the GUI.For example, GUI updater 178 prepares display information and sends itto graphics module 132 for display on a touch-sensitive display.

In some embodiments, event handler(s) 190 includes or has access to dataupdater 176, object updater 177, and GUI updater 178. In someembodiments, data updater 176, object updater 177, and GUI updater 178are included in a single module of a respective application 136-1 orapplication view 191. In other embodiments, they are included in two ormore software modules.

It shall be understood that the foregoing discussion regarding eventhandling of user touches on touch-sensitive displays also applies toother forms of user inputs to operate multifunction devices 100 withinput-devices, not all of which are initiated on touch screens. Forexample, mouse movement and mouse button presses, optionally coordinatedwith single or multiple keyboard presses or holds; contact movementssuch as taps, drags, scrolls, etc., on touch-pads; pen stylus inputs;movement of the device; oral instructions; detected eye movements;biometric inputs; and/or any combination thereof are optionally utilizedas inputs corresponding to sub-events which define an event to berecognized.

FIG. 1C is a block diagram illustrating a tactile output module inaccordance with some embodiments. In some embodiments, I/O subsystem 106(e.g., haptic feedback controller 161 (FIG. 1A) and/or other inputcontroller(s) 160 (FIG. 1A)) includes at least some of the examplecomponents shown in FIG. 1C. In some embodiments, peripherals interface118 includes at least some of the example components shown in FIG. 1C.

In some embodiments, the tactile output module includes haptic feedbackmodule 133. In some embodiments, haptic feedback module 133 aggregatesand combines tactile outputs for user interface feedback from softwareapplications on the electronic device (e.g., feedback that is responsiveto user inputs that correspond to displayed user interfaces and alertsand other notifications that indicate the performance of operations oroccurrence of events in user interfaces of the electronic device).Haptic feedback module 133 includes one or more of: waveform module 123(for providing waveforms used for generating tactile outputs), mixer 125(for mixing waveforms, such as waveforms in different channels),compressor 127 (for reducing or compressing a dynamic range of thewaveforms), low-pass filter 129 (for filtering out high frequency signalcomponents in the waveforms), and thermal controller 131 (for adjustingthe waveforms in accordance with thermal conditions). In someembodiments, haptic feedback module 133 is included in haptic feedbackcontroller 161 (FIG. 1A). In some embodiments, a separate unit of hapticfeedback module 133 (or a separate implementation of haptic feedbackmodule 133) is also included in an audio controller (e.g., audiocircuitry 110, FIG. 1A) and used for generating audio signals. In someembodiments, a single haptic feedback module 133 is used for generatingaudio signals and generating waveforms for tactile outputs.

In some embodiments, haptic feedback module 133 also includes triggermodule 121 (e.g., a software application, operating system, or othersoftware module that determines a tactile output is to be generated andinitiates the process for generating the corresponding tactile output).In some embodiments, trigger module 121 generates trigger signals forinitiating generation of waveforms (e.g., by waveform module 123). Forexample, trigger module 121 generates trigger signals based on presettiming criteria. In some embodiments, trigger module 121 receivestrigger signals from outside haptic feedback module 133 (e.g., in someembodiments, haptic feedback module 133 receives trigger signals fromhardware input processing module 146 located outside haptic feedbackmodule 133) and relays the trigger signals to other components withinhaptic feedback module 133 (e.g., waveform module 123) or softwareapplications that trigger operations (e.g., with trigger module 121)based on activation of a user interface element (e.g., an applicationicon or an affordance within an application) or a hardware input device(e.g., a home button or an intensity-sensitive input surface, such as anintensity-sensitive touch screen). In some embodiments, trigger module121 also receives tactile feedback generation instructions (e.g., fromhaptic feedback module 133, FIGS. 1A and 3). In some embodiments,trigger module 121 generates trigger signals in response to hapticfeedback module 133 (or trigger module 121 in haptic feedback module133) receiving tactile feedback instructions (e.g., from haptic feedbackmodule 133, FIGS. 1A and 3).

Waveform module 123 receives trigger signals (e.g., from trigger module121) as an input, and in response to receiving trigger signals, provideswaveforms for generation of one or more tactile outputs (e.g., waveformsselected from a predefined set of waveforms designated for use bywaveform module 123, such as the waveforms described in greater detailbelow with reference to FIGS. 4F-4G).

Mixer 125 receives waveforms (e.g., from waveform module 123) as aninput, and mixes together the waveforms. For example, when mixer 125receives two or more waveforms (e.g., a first waveform in a firstchannel and a second waveform that at least partially overlaps with thefirst waveform in a second channel) mixer 125 outputs a combinedwaveform that corresponds to a sum of the two or more waveforms. In someembodiments, mixer 125 also modifies one or more waveforms of the two ormore waveforms to emphasize particular waveform(s) over the rest of thetwo or more waveforms (e.g., by increasing a scale of the particularwaveform(s) and/or decreasing a scale of the rest of the waveforms). Insome circumstances, mixer 125 selects one or more waveforms to removefrom the combined waveform (e.g., the waveform from the oldest source isdropped when there are waveforms from more than three sources that havebeen requested to be output concurrently by tactile output generator167).

Compressor 127 receives waveforms (e.g., a combined waveform from mixer125) as an input, and modifies the waveforms. In some embodiments,compressor 127 reduces the waveforms (e.g., in accordance with physicalspecifications of tactile output generators 167 (FIG. 1A) or 357 (FIG.3)) so that tactile outputs corresponding to the waveforms are reduced.In some embodiments, compressor 127 limits the waveforms, such as byenforcing a predefined maximum amplitude for the waveforms. For example,compressor 127 reduces amplitudes of portions of waveforms that exceed apredefined amplitude threshold while maintaining amplitudes of portionsof waveforms that do not exceed the predefined amplitude threshold. Insome embodiments, compressor 127 reduces a dynamic range of thewaveforms. In some embodiments, compressor 127 dynamically reduces thedynamic range of the waveforms so that the combined waveforms remainwithin performance specifications of the tactile output generator 167(e.g., force and/or moveable mass displacement limits).

Low-pass filter 129 receives waveforms (e.g., compressed waveforms fromcompressor 127) as an input, and filters (e.g., smooths) the waveforms(e.g., removes or reduces high frequency signal components in thewaveforms). For example, in some instances, compressor 127 includes, incompressed waveforms, extraneous signals (e.g., high frequency signalcomponents) that interfere with the generation of tactile outputs and/orexceed performance specifications of tactile output generator 167 whenthe tactile outputs are generated in accordance with the compressedwaveforms. Low-pass filter 129 reduces or removes such extraneoussignals in the waveforms.

Thermal controller 131 receives waveforms (e.g., filtered waveforms fromlow-pass filter 129) as an input, and adjusts the waveforms inaccordance with thermal conditions of device 100 (e.g., based oninternal temperatures detected within device 100, such as thetemperature of haptic feedback controller 161, and/or externaltemperatures detected by device 100). For example, in some cases, theoutput of haptic feedback controller 161 varies depending on thetemperature (e.g. haptic feedback controller 161, in response toreceiving same waveforms, generates a first tactile output when hapticfeedback controller 161 is at a first temperature and generates a secondtactile output when haptic feedback controller 161 is at a secondtemperature that is distinct from the first temperature). For example,the magnitude (or the amplitude) of the tactile outputs may varydepending on the temperature. To reduce the effect of the temperaturevariations, the waveforms are modified (e.g., an amplitude of thewaveforms is increased or decreased based on the temperature).

In some embodiments, haptic feedback module 133 (e.g., trigger module121) is coupled to hardware input processing module 146. In someembodiments, other input controller(s) 160 in FIG. 1A includes hardwareinput processing module 146. In some embodiments, hardware inputprocessing module 146 receives inputs from hardware input device 145(e.g., other input or control devices 116 in FIG. 1A, such as a homebutton or an intensity-sensitive input surface, such as anintensity-sensitive touch screen). In some embodiments, hardware inputdevice 145 is any input device described herein, such as touch-sensitivedisplay system 112 (FIG. 1A), keyboard/mouse 350 (FIG. 3), touchpad 355(FIG. 3), one of other input or control devices 116 (FIG. 1A), or anintensity-sensitive home button. In some embodiments, hardware inputdevice 145 consists of an intensity-sensitive home button, and nottouch-sensitive display system 112 (FIG. 1A), keyboard/mouse 350 (FIG.3), or touchpad 355 (FIG. 3). In some embodiments, in response to inputsfrom hardware input device 145 (e.g., an intensity-sensitive home buttonor a touch screen), hardware input processing module 146 provides one ormore trigger signals to haptic feedback module 133 to indicate that auser input satisfying predefined input criteria, such as an inputcorresponding to a “click” of a home button (e.g., a “down click” or an“up click”), has been detected. In some embodiments, haptic feedbackmodule 133 provides waveforms that correspond to the “click” of a homebutton in response to the input corresponding to the “click” of a homebutton, simulating a haptic feedback of pressing a physical home button.

In some embodiments, the tactile output module includes haptic feedbackcontroller 161 (e.g., haptic feedback controller 161 in FIG. 1A), whichcontrols the generation of tactile outputs. In some embodiments, hapticfeedback controller 161 is coupled to a plurality of tactile outputgenerators, and selects one or more tactile output generators of theplurality of tactile output generators and sends waveforms to theselected one or more tactile output generators for generating tactileoutputs. In some embodiments, haptic feedback controller 161 coordinatestactile output requests that correspond to activation of hardware inputdevice 145 and tactile output requests that correspond to softwareevents (e.g., tactile output requests from haptic feedback module 133)and modifies one or more waveforms of the two or more waveforms toemphasize particular waveform(s) over the rest of the two or morewaveforms (e.g., by increasing a scale of the particular waveform(s)and/or decreasing a scale of the rest of the waveforms, such as toprioritize tactile outputs that correspond to activations of hardwareinput device 145 over tactile outputs that correspond to softwareevents).

In some embodiments, as shown in FIG. 1C, an output of haptic feedbackcontroller 161 is coupled to audio circuitry of device 100 (e.g., audiocircuitry 110, FIG. 1A), and provides audio signals to audio circuitryof device 100. In some embodiments, haptic feedback controller 161provides both waveforms used for generating tactile outputs and audiosignals used for providing audio outputs in conjunction with generationof the tactile outputs. In some embodiments, haptic feedback controller161 modifies audio signals and/or waveforms (used for generating tactileoutputs) so that the audio outputs and the tactile outputs aresynchronized (e.g., by delaying the audio signals and/or waveforms). Insome embodiments, haptic feedback controller 161 includes adigital-to-analog converter used for converting digital waveforms intoanalog signals, which are received by amplifier 163 and/or tactileoutput generator 167.

In some embodiments, the tactile output module includes amplifier 163.In some embodiments, amplifier 163 receives waveforms (e.g., from hapticfeedback controller 161) and amplifies the waveforms prior to sendingthe amplified waveforms to tactile output generator 167 (e.g., any oftactile output generators 167 (FIG. 1A) or 357 (FIG. 3)). For example,amplifier 163 amplifies the received waveforms to signal levels that arein accordance with physical specifications of tactile output generator167 (e.g., to a voltage and/or a current required by tactile outputgenerator 167 for generating tactile outputs so that the signals sent totactile output generator 167 produce tactile outputs that correspond tothe waveforms received from haptic feedback controller 161) and sendsthe amplified waveforms to tactile output generator 167. In response,tactile output generator 167 generates tactile outputs (e.g., byshifting a moveable mass back and forth in one or more dimensionsrelative to a neutral position of the moveable mass).

In some embodiments, the tactile output module includes sensor 169,which is coupled to tactile output generator 167. Sensor 169 detectsstates or state changes (e.g., mechanical position, physicaldisplacement, and/or movement) of tactile output generator 167 or one ormore components of tactile output generator 167 (e.g., one or moremoving parts, such as a membrane, used to generate tactile outputs). Insome embodiments, sensor 169 is a magnetic field sensor (e.g., a Halleffect sensor) or other displacement and/or movement sensor. In someembodiments, sensor 169 provides information (e.g., a position, adisplacement, and/or a movement of one or more parts in tactile outputgenerator 167) to haptic feedback controller 161 and, in accordance withthe information provided by sensor 169 about the state of tactile outputgenerator 167, haptic feedback controller 161 adjusts the waveformsoutput from haptic feedback controller 161 (e.g., waveforms sent totactile output generator 167, optionally via amplifier 163).

FIG. 2 illustrates a portable multifunction device 100 having a touchscreen (e.g., touch-sensitive display system 112, FIG. 1A) in accordancewith some embodiments. The touch screen optionally displays one or moregraphics within user interface (UI) 200. In these embodiments, as wellas others described below, a user is enabled to select one or more ofthe graphics by making a gesture on the graphics, for example, with oneor more fingers 202 (not drawn to scale in the figure) or one or morestyluses 203 (not drawn to scale in the figure). In some embodiments,selection of one or more graphics occurs when the user breaks contactwith the one or more graphics. In some embodiments, the gestureoptionally includes one or more taps, one or more swipes (from left toright, right to left, upward and/or downward) and/or a rolling of afinger (from right to left, left to right, upward and/or downward) thathas made contact with device 100. In some implementations orcircumstances, inadvertent contact with a graphic does not select thegraphic. For example, a swipe gesture that sweeps over an applicationicon optionally does not select the corresponding application when thegesture corresponding to selection is a tap.

Device 100 optionally also includes one or more physical buttons, suchas “home” or menu button 204. As described previously, menu button 204is, optionally, used to navigate to any application 136 in a set ofapplications that are, optionally executed on device 100. Alternatively,in some embodiments, the menu button is implemented as a soft key in aGUI displayed on the touch-screen display.

In some embodiments, device 100 includes the touch-screen display, menubutton 204 (sometimes called home button 204), push button 206 forpowering the device on/off and locking the device, volume adjustmentbutton(s) 208, Subscriber Identity Module (SIM) card slot 210, head setjack 212, and docking/charging external port 124. Push button 206 is,optionally, used to turn the power on/off on the device by depressingthe button and holding the button in the depressed state for apredefined time interval; to lock the device by depressing the buttonand releasing the button before the predefined time interval haselapsed; and/or to unlock the device or initiate an unlock process. Insome embodiments, device 100 also accepts verbal input for activation ordeactivation of some functions through microphone 113. Device 100 also,optionally, includes one or more contact intensity sensors 165 fordetecting intensities of contacts on touch-sensitive display system 112and/or one or more tactile output generators 167 for generating tactileoutputs for a user of device 100.

FIG. 3 is a block diagram of an example multifunction device with adisplay and a touch-sensitive surface in accordance with someembodiments. Device 300 need not be portable. In some embodiments,device 300 is a laptop computer, a desktop computer, a tablet computer,a multimedia player device, a navigation device, an educational device(such as a child's learning toy), a gaming system, or a control device(e.g., a home or industrial controller). Device 300 typically includesone or more processing units (CPU's) 310, one or more network or othercommunications interfaces 360, memory 370, and one or more communicationbuses 320 for interconnecting these components. Communication buses 320optionally include circuitry (sometimes called a chipset) thatinterconnects and controls communications between system components.Device 300 includes input/output (I/O) interface 330 comprising display340, which is typically a touch-screen display. I/O interface 330 alsooptionally includes a keyboard and/or mouse (or other pointing device)350 and touchpad 355, tactile output generator 357 for generatingtactile outputs on device 300 (e.g., similar to tactile outputgenerator(s) 167 described above with reference to FIG. 1A), sensors 359(e.g., optical, acceleration, proximity, touch-sensitive, and/or contactintensity sensors similar to contact intensity sensor(s) 165 describedabove with reference to FIG. 1A). Memory 370 includes high-speed randomaccess memory, such as DRAM, SRAM, DDR RAM or other random access solidstate memory devices; and optionally includes non-volatile memory, suchas one or more magnetic disk storage devices, optical disk storagedevices, flash memory devices, or other non-volatile solid state storagedevices. Memory 370 optionally includes one or more storage devicesremotely located from CPU(s) 310. In some embodiments, memory 370 storesprograms, modules, and data structures analogous to the programs,modules, and data structures stored in memory 102 of portablemultifunction device 100 (FIG. 1A), or a subset thereof. Furthermore,memory 370 optionally stores additional programs, modules, and datastructures not present in memory 102 of portable multifunction device100. For example, memory 370 of device 300 optionally stores drawingmodule 380, presentation module 382, word processing module 384, websitecreation module 386, disk authoring module 388, and/or spreadsheetmodule 390, while memory 102 of portable multifunction device 100 (FIG.1A) optionally does not store these modules.

Each of the above identified elements in FIG. 3 are, optionally, storedin one or more of the previously mentioned memory devices. Each of theabove identified modules corresponds to a set of instructions forperforming a function described above. The above identified modules orprograms (i.e., sets of instructions) need not be implemented asseparate software programs, procedures or modules, and thus varioussubsets of these modules are, optionally, combined or otherwisere-arranged in various embodiments. In some embodiments, memory 370optionally stores a subset of the modules and data structures identifiedabove. Furthermore, memory 370 optionally stores additional modules anddata structures not described above.

Attention is now directed towards embodiments of user interfaces (“UI”)that are, optionally, implemented on portable multifunction device 100.

FIG. 4A illustrates an example user interface 400 for a menu ofapplications on portable multifunction device 100 in accordance withsome embodiments. Similar user interfaces are, optionally, implementedon device 300. In some embodiments, user interface 400 includes thefollowing elements, or a subset or superset thereof:

-   -   Signal strength indicator(s) for wireless communication(s), such        as cellular and Wi-Fi signals;    -   Time;    -   a Bluetooth indicator;    -   a Battery status indicator;    -   Tray 408 with icons for frequently used applications, such as:        -   Icon 416 for telephone module 138, labeled “Phone,” which            optionally includes an indicator 414 of the number of missed            calls or voicemail messages;        -   Icon 418 for e-mail client module 140, labeled “Mail,” which            optionally includes an indicator 410 of the number of unread            e-mails;        -   Icon 420 for browser module 147, labeled “Browser;” and        -   Icon 422 for video and music player module 152, labeled            “Music;” and    -   Icons for other applications, such as:        -   Icon 424 for IM module 141, labeled “Messages;”        -   Icon 426 for calendar module 148, labeled “Calendar;”        -   Icon 428 for image management module 144, labeled “Photos;”        -   Icon 430 for camera module 143, labeled “Camera;”        -   Icon 432 for online video module 155, labeled “Online            Video;”        -   Icon 434 for stocks widget 149-2, labeled “Stocks;”        -   Icon 436 for map module 154, labeled “Maps;”        -   Icon 438 for weather widget 149-1, labeled “Weather;”        -   Icon 440 for alarm clock widget 149-4, labeled “Clock;”        -   Icon 442 for workout support module 142, labeled “Workout            Support;”        -   Icon 444 for notes module 153, labeled “Notes;” and        -   Icon 446 for a settings application or module, which            provides access to settings for device 100 and its various            applications 136.

It should be noted that the icon labels illustrated in FIG. 4A aremerely examples. For example, other labels are, optionally, used forvarious application icons. In some embodiments, a label for a respectiveapplication icon includes a name of an application corresponding to therespective application icon. In some embodiments, a label for aparticular application icon is distinct from a name of an applicationcorresponding to the particular application icon.

FIG. 4B illustrates an example user interface on a device (e.g., device300, FIG. 3) with a touch-sensitive surface 451 (e.g., a tablet ortouchpad 355, FIG. 3) that is separate from the display 450. Althoughmany of the examples that follow will be given with reference to inputson touch screen touch-sensitive display system 112 (where the touchsensitive surface and the display are combined), in some embodiments,the device detects inputs on a touch-sensitive surface that is separatefrom the display, as shown in FIG. 4B. In some embodiments, thetouch-sensitive surface (e.g., 451 in FIG. 4B) has a primary axis (e.g.,452 in FIG. 4B) that corresponds to a primary axis (e.g., 453 in FIG.4B) on the display (e.g., 450). In accordance with these embodiments,the device detects contacts (e.g., 460 and 462 in FIG. 4B) with thetouch-sensitive surface 451 at locations that correspond to respectivelocations on the display (e.g., in FIG. 4B, 460 corresponds to 468 and462 corresponds to 470). In this way, user inputs (e.g., contacts 460and 462, and movements thereof) detected by the device on thetouch-sensitive surface (e.g., 451 in FIG. 4B) are used by the device tomanipulate the user interface on the display (e.g., 450 in FIG. 4B) ofthe multifunction device when the touch-sensitive surface is separatefrom the display. It should be understood that similar methods are,optionally, used for other user interfaces described herein.

Additionally, while the following examples are given primarily withreference to finger inputs (e.g., finger contacts, finger tap gestures,finger swipe gestures, etc.), it should be understood that, in someembodiments, one or more of the finger inputs are replaced with inputfrom another input device (e.g., a mouse based input or a stylus input).For example, a swipe gesture is, optionally, replaced with a mouse click(e.g., instead of a contact) followed by movement of the cursor alongthe path of the swipe (e.g., instead of movement of the contact). Asanother example, a tap gesture is, optionally, replaced with a mouseclick while the cursor is located over the location of the tap gesture(e.g., instead of detection of the contact followed by ceasing to detectthe contact). Similarly, when multiple user inputs are simultaneouslydetected, it should be understood that multiple computer mice are,optionally, used simultaneously, or a mouse and finger contacts are,optionally, used simultaneously.

As used herein, the term “focus selector” refers to an input elementthat indicates a current part of a user interface with which a user isinteracting. In some implementations that include a cursor or otherlocation marker, the cursor acts as a “focus selector,” so that when aninput (e.g., a press input) is detected on a touch-sensitive surface(e.g., touchpad 355 in FIG. 3 or touch-sensitive surface 451 in FIG. 4B)while the cursor is over a particular user interface element (e.g., abutton, window, slider or other user interface element), the particularuser interface element is adjusted in accordance with the detectedinput. In some implementations that include a touch-screen display(e.g., touch-sensitive display system 112 in FIG. 1A or the touch screenin FIG. 4A) that enables direct interaction with user interface elementson the touch-screen display, a detected contact on the touch-screen actsas a “focus selector,” so that when an input (e.g., a press input by thecontact) is detected on the touch-screen display at a location of aparticular user interface element (e.g., a button, window, slider orother user interface element), the particular user interface element isadjusted in accordance with the detected input. In some implementations,focus is moved from one region of a user interface to another region ofthe user interface without corresponding movement of a cursor ormovement of a contact on a touch-screen display (e.g., by using a tabkey or arrow keys to move focus from one button to another button); inthese implementations, the focus selector moves in accordance withmovement of focus between different regions of the user interface.Without regard to the specific form taken by the focus selector, thefocus selector is generally the user interface element (or contact on atouch-screen display) that is controlled by the user so as tocommunicate the user's intended interaction with the user interface(e.g., by indicating, to the device, the element of the user interfacewith which the user is intending to interact). For example, the locationof a focus selector (e.g., a cursor, a contact, or a selection box) overa respective button while a press input is detected on thetouch-sensitive surface (e.g., a touchpad or touch screen) will indicatethat the user is intending to activate the respective button (as opposedto other user interface elements shown on a display of the device).

As used in the specification and claims, the term “intensity” of acontact on a touch-sensitive surface refers to the force or pressure(force per unit area) of a contact (e.g., a finger contact or a styluscontact) on the touch-sensitive surface, or to a substitute (proxy) forthe force or pressure of a contact on the touch-sensitive surface. Theintensity of a contact has a range of values that includes at least fourdistinct values and more typically includes hundreds of distinct values(e.g., at least 256). Intensity of a contact is, optionally, determined(or measured) using various approaches and various sensors orcombinations of sensors. For example, one or more force sensorsunderneath or adjacent to the touch-sensitive surface are, optionally,used to measure force at various points on the touch-sensitive surface.In some implementations, force measurements from multiple force sensorsare combined (e.g., a weighted average or a sum) to determine anestimated force of a contact. Similarly, a pressure-sensitive tip of astylus is, optionally, used to determine a pressure of the stylus on thetouch-sensitive surface. Alternatively, the size of the contact areadetected on the touch-sensitive surface and/or changes thereto, thecapacitance of the touch-sensitive surface proximate to the contactand/or changes thereto, and/or the resistance of the touch-sensitivesurface proximate to the contact and/or changes thereto are, optionally,used as a substitute for the force or pressure of the contact on thetouch-sensitive surface. In some implementations, the substitutemeasurements for contact force or pressure are used directly todetermine whether an intensity threshold has been exceeded (e.g., theintensity threshold is described in units corresponding to thesubstitute measurements). In some implementations, the substitutemeasurements for contact force or pressure are converted to an estimatedforce or pressure and the estimated force or pressure is used todetermine whether an intensity threshold has been exceeded (e.g., theintensity threshold is a pressure threshold measured in units ofpressure). Using the intensity of a contact as an attribute of a userinput allows for user access to additional device functionality that mayotherwise not be readily accessible by the user on a reduced-size devicewith limited real estate for displaying affordances (e.g., on atouch-sensitive display) and/or receiving user input (e.g., via atouch-sensitive display, a touch-sensitive surface, or aphysical/mechanical control such as a knob or a button).

In some embodiments, contact/motion module 130 uses a set of one or moreintensity thresholds to determine whether an operation has beenperformed by a user (e.g., to determine whether a user has “clicked” onan icon). In some embodiments, at least a subset of the intensitythresholds is determined in accordance with software parameters (e.g.,the intensity thresholds are not determined by the activation thresholdsof particular physical actuators and can be adjusted without changingthe physical hardware of device 100). For example, a mouse “click”threshold of a trackpad or touch-screen display can be set to any of alarge range of predefined thresholds values without changing thetrackpad or touch-screen display hardware. Additionally, in someimplementations a user of the device is provided with software settingsfor adjusting one or more of the set of intensity thresholds (e.g., byadjusting individual intensity thresholds and/or by adjusting aplurality of intensity thresholds at once with a system-level click“intensity” parameter).

As used in the specification and claims, the term “characteristicintensity” of a contact refers to a characteristic of the contact basedon one or more intensities of the contact. In some embodiments, thecharacteristic intensity is based on multiple intensity samples. Thecharacteristic intensity is, optionally, based on a predefined number ofintensity samples, or a set of intensity samples collected during apredetermined time period (e.g., 0.05, 0.1, 0.2, 0.5, 1, 2, 5, 10seconds) relative to a predefined event (e.g., after detecting thecontact, prior to detecting liftoff of the contact, before or afterdetecting a start of movement of the contact, prior to detecting an endof the contact, before or after detecting an increase in intensity ofthe contact, and/or before or after detecting a decrease in intensity ofthe contact). A characteristic intensity of a contact is, optionallybased on one or more of: a maximum value of the intensities of thecontact, a mean value of the intensities of the contact, an averagevalue of the intensities of the contact, a top 10 percentile value ofthe intensities of the contact, a value at the half maximum of theintensities of the contact, a value at the 90 percent maximum of theintensities of the contact, a value produced by low-pass filtering theintensity of the contact over a predefined period or starting at apredefined time, or the like. In some embodiments, the duration of thecontact is used in determining the characteristic intensity (e.g., whenthe characteristic intensity is an average of the intensity of thecontact over time). In some embodiments, the characteristic intensity iscompared to a set of one or more intensity thresholds to determinewhether an operation has been performed by a user. For example, the setof one or more intensity thresholds may include a first intensitythreshold and a second intensity threshold. In this example, a contactwith a characteristic intensity that does not exceed the first intensitythreshold results in a first operation, a contact with a characteristicintensity that exceeds the first intensity threshold and does not exceedthe second intensity threshold results in a second operation, and acontact with a characteristic intensity that exceeds the secondintensity threshold results in a third operation. In some embodiments, acomparison between the characteristic intensity and one or moreintensity thresholds is used to determine whether or not to perform oneor more operations (e.g., whether to perform a respective option orforgo performing the respective operation) rather than being used todetermine whether to perform a first operation or a second operation.

In some embodiments, a portion of a gesture is identified for purposesof determining a characteristic intensity. For example, atouch-sensitive surface may receive a continuous swipe contacttransitioning from a start location and reaching an end location (e.g.,a drag gesture), at which point the intensity of the contact increases.In this example, the characteristic intensity of the contact at the endlocation may be based on only a portion of the continuous swipe contact,and not the entire swipe contact (e.g., only the portion of the swipecontact at the end location). In some embodiments, a smoothing algorithmmay be applied to the intensities of the swipe contact prior todetermining the characteristic intensity of the contact. For example,the smoothing algorithm optionally includes one or more of: anunweighted sliding-average smoothing algorithm, a triangular smoothingalgorithm, a median filter smoothing algorithm, and/or an exponentialsmoothing algorithm. In some circumstances, these smoothing algorithmseliminate narrow spikes or dips in the intensities of the swipe contactfor purposes of determining a characteristic intensity.

The user interface figures described herein optionally include variousintensity diagrams that show the current intensity of the contact on thetouch-sensitive surface relative to one or more intensity thresholds(e.g., a contact detection intensity threshold IT₀, a light pressintensity threshold IT_(L), a deep press intensity threshold IT_(D)(e.g., that is at least initially higher than IT_(L)), and/or one ormore other intensity thresholds (e.g., an intensity threshold IT_(H)that is lower than IT_(L))). This intensity diagram is typically notpart of the displayed user interface, but is provided to aid in theinterpretation of the figures. In some embodiments, the light pressintensity threshold corresponds to an intensity at which the device willperform operations typically associated with clicking a button of aphysical mouse or a trackpad. In some embodiments, the deep pressintensity threshold corresponds to an intensity at which the device willperform operations that are different from operations typicallyassociated with clicking a button of a physical mouse or a trackpad. Insome embodiments, when a contact is detected with a characteristicintensity below the light press intensity threshold (e.g., and above anominal contact-detection intensity threshold IT₀ below which thecontact is no longer detected), the device will move a focus selector inaccordance with movement of the contact on the touch-sensitive surfacewithout performing an operation associated with the light pressintensity threshold or the deep press intensity threshold. Generally,unless otherwise stated, these intensity thresholds are consistentbetween different sets of user interface figures.

In some embodiments, the response of the device to inputs detected bythe device depends on criteria based on the contact intensity during theinput. For example, for some “light press” inputs, the intensity of acontact exceeding a first intensity threshold during the input triggersa first response. In some embodiments, the response of the device toinputs detected by the device depends on criteria that include both thecontact intensity during the input and time-based criteria. For example,for some “deep press” inputs, the intensity of a contact exceeding asecond intensity threshold during the input, greater than the firstintensity threshold for a light press, triggers a second response onlyif a delay time has elapsed between meeting the first intensitythreshold and meeting the second intensity threshold. This delay time istypically less than 200 ms (milliseconds) in duration (e.g., 40, 100, or120 ms, depending on the magnitude of the second intensity threshold,with the delay time increasing as the second intensity thresholdincreases). This delay time helps to avoid accidental recognition ofdeep press inputs. As another example, for some “deep press” inputs,there is a reduced-sensitivity time period that occurs after the time atwhich the first intensity threshold is met. During thereduced-sensitivity time period, the second intensity threshold isincreased. This temporary increase in the second intensity thresholdalso helps to avoid accidental deep press inputs. For other deep pressinputs, the response to detection of a deep press input does not dependon time-based criteria.

In some embodiments, one or more of the input intensity thresholdsand/or the corresponding outputs vary based on one or more factors, suchas user settings, contact motion, input timing, application running,rate at which the intensity is applied, number of concurrent inputs,user history, environmental factors (e.g., ambient noise), focusselector position, and the like. Example factors are described in U.S.patent application Ser. Nos. 14/399,606 and 14/624,296, which areincorporated by reference herein in their entireties.

For example, FIG. 4C illustrates a dynamic intensity threshold 480 thatchanges over time based in part on the intensity of touch input 476 overtime. Dynamic intensity threshold 480 is a sum of two components, firstcomponent 474 that decays over time after a predefined delay time p1from when touch input 476 is initially detected, and second component478 that trails the intensity of touch input 476 over time. The initialhigh intensity threshold of first component 474 reduces accidentaltriggering of a “deep press” response, while still allowing an immediate“deep press” response if touch input 476 provides sufficient intensity.Second component 478 reduces unintentional triggering of a “deep press”response by gradual intensity fluctuations of in a touch input. In someembodiments, when touch input 476 satisfies dynamic intensity threshold480 (e.g., at point 481 in FIG. 4C), the “deep press” response istriggered.

FIG. 4D illustrates another dynamic intensity threshold 486 (e.g.,intensity threshold I_(D)). FIG. 4D also illustrates two other intensitythresholds: a first intensity threshold I_(H) and a second intensitythreshold I_(L). In FIG. 4D, although touch input 484 satisfies thefirst intensity threshold I_(H) and the second intensity threshold I_(L)prior to time p2, no response is provided until delay time p2 haselapsed at time 482. Also in FIG. 4D, dynamic intensity threshold 486decays over time, with the decay starting at time 488 after a predefineddelay time p1 has elapsed from time 482 (when the response associatedwith the second intensity threshold I_(L) was triggered). This type ofdynamic intensity threshold reduces accidental triggering of a responseassociated with the dynamic intensity threshold I_(D) immediately after,or concurrently with, triggering a response associated with a lowerintensity threshold, such as the first intensity threshold I_(H) or thesecond intensity threshold I_(L).

FIG. 4E illustrate yet another dynamic intensity threshold 492 (e.g.,intensity threshold I_(D)). In FIG. 4E, a response associated with theintensity threshold I_(L) is triggered after the delay time p2 haselapsed from when touch input 490 is initially detected. Concurrently,dynamic intensity threshold 492 decays after the predefined delay timep1 has elapsed from when touch input 490 is initially detected. So adecrease in intensity of touch input 490 after triggering the responseassociated with the intensity threshold I_(L), followed by an increasein the intensity of touch input 490, without releasing touch input 490,can trigger a response associated with the intensity threshold I_(D)(e.g., at time 494) even when the intensity of touch input 490 is belowanother intensity threshold, for example, the intensity threshold I_(L).

An increase of characteristic intensity of the contact from an intensitybelow the light press intensity threshold IT_(L) to an intensity betweenthe light press intensity threshold IT_(L) and the deep press intensitythreshold IT_(D) is sometimes referred to as a “light press” input. Anincrease of characteristic intensity of the contact from an intensitybelow the deep press intensity threshold ITS to an intensity above thedeep press intensity threshold IT_(D) is sometimes referred to as a“deep press” input. An increase of characteristic intensity of thecontact from an intensity below the contact-detection intensitythreshold IT₀ to an intensity between the contact-detection intensitythreshold IT₀ and the light press intensity threshold IT_(L) issometimes referred to as detecting the contact on the touch-surface. Adecrease of characteristic intensity of the contact from an intensityabove the contact-detection intensity threshold IT₀ to an intensitybelow the contact-detection intensity threshold IT₀ is sometimesreferred to as detecting liftoff of the contact from the touch-surface.In some embodiments IT₀ is zero. In some embodiments, IT₀ is greaterthan zero. In some illustrations a shaded circle or oval is used torepresent intensity of a contact on the touch-sensitive surface. In someillustrations, a circle or oval without shading is used represent arespective contact on the touch-sensitive surface without specifying theintensity of the respective contact.

In some embodiments, described herein, one or more operations areperformed in response to detecting a gesture that includes a respectivepress input or in response to detecting the respective press inputperformed with a respective contact (or a plurality of contacts), wherethe respective press input is detected based at least in part ondetecting an increase in intensity of the contact (or plurality ofcontacts) above a press-input intensity threshold. In some embodiments,the respective operation is performed in response to detecting theincrease in intensity of the respective contact above the press-inputintensity threshold (e.g., the respective operation is performed on a“down stroke” of the respective press input). In some embodiments, thepress input includes an increase in intensity of the respective contactabove the press-input intensity threshold and a subsequent decrease inintensity of the contact below the press-input intensity threshold, andthe respective operation is performed in response to detecting thesubsequent decrease in intensity of the respective contact below thepress-input threshold (e.g., the respective operation is performed on an“up stroke” of the respective press input).

In some embodiments, the device employs intensity hysteresis to avoidaccidental inputs sometimes termed “jitter,” where the device defines orselects a hysteresis intensity threshold with a predefined relationshipto the press-input intensity threshold (e.g., the hysteresis intensitythreshold is X intensity units lower than the press-input intensitythreshold or the hysteresis intensity threshold is 75%, 90%, or somereasonable proportion of the press-input intensity threshold). Thus, insome embodiments, the press input includes an increase in intensity ofthe respective contact above the press-input intensity threshold and asubsequent decrease in intensity of the contact below the hysteresisintensity threshold that corresponds to the press-input intensitythreshold, and the respective operation is performed in response todetecting the subsequent decrease in intensity of the respective contactbelow the hysteresis intensity threshold (e.g., the respective operationis performed on an “up stroke” of the respective press input).Similarly, in some embodiments, the press input is detected only whenthe device detects an increase in intensity of the contact from anintensity at or below the hysteresis intensity threshold to an intensityat or above the press-input intensity threshold and, optionally, asubsequent decrease in intensity of the contact to an intensity at orbelow the hysteresis intensity, and the respective operation isperformed in response to detecting the press input (e.g., the increasein intensity of the contact or the decrease in intensity of the contact,depending on the circumstances).

For ease of explanation, the description of operations performed inresponse to a press input associated with a press-input intensitythreshold or in response to a gesture including the press input are,optionally, triggered in response to detecting: an increase in intensityof a contact above the press-input intensity threshold, an increase inintensity of a contact from an intensity below the hysteresis intensitythreshold to an intensity above the press-input intensity threshold, adecrease in intensity of the contact below the press-input intensitythreshold, or a decrease in intensity of the contact below thehysteresis intensity threshold corresponding to the press-inputintensity threshold. Additionally, in examples where an operation isdescribed as being performed in response to detecting a decrease inintensity of a contact below the press-input intensity threshold, theoperation is, optionally, performed in response to detecting a decreasein intensity of the contact below a hysteresis intensity thresholdcorresponding to, and lower than, the press-input intensity threshold.As described above, in some embodiments, the triggering of theseresponses also depends on time-based criteria being met (e.g., a delaytime has elapsed between a first intensity threshold being met and asecond intensity threshold being met).

As used in the specification and claims, the term “tactile output”refers to physical displacement of a device relative to a previousposition of the device, physical displacement of a component (e.g., atouch-sensitive surface) of a device relative to another component(e.g., housing) of the device, or displacement of the component relativeto a center of mass of the device that will be detected by a user withthe user's sense of touch. For example, in situations where the deviceor the component of the device is in contact with a surface of a userthat is sensitive to touch (e.g., a finger, palm, or other part of auser's hand), the tactile output generated by the physical displacementwill be interpreted by the user as a tactile sensation corresponding toa perceived change in physical characteristics of the device or thecomponent of the device. For example, movement of a touch-sensitivesurface (e.g., a touch-sensitive display or trackpad) is, optionally,interpreted by the user as a “down click” or “up click” of a physicalactuator button. In some cases, a user will feel a tactile sensationsuch as an “down click” or “up click” even when there is no movement ofa physical actuator button associated with the touch-sensitive surfacethat is physically pressed (e.g., displaced) by the user's movements. Asanother example, movement of the touch-sensitive surface is, optionally,interpreted or sensed by the user as “roughness” of the touch-sensitivesurface, even when there is no change in smoothness of thetouch-sensitive surface. While such interpretations of touch by a userwill be subject to the individualized sensory perceptions of the user,there are many sensory perceptions of touch that are common to a largemajority of users. Thus, when a tactile output is described ascorresponding to a particular sensory perception of a user (e.g., an “upclick,” a “down click,” “roughness”), unless otherwise stated, thegenerated tactile output corresponds to physical displacement of thedevice or a component thereof that will generate the described sensoryperception for a typical (or average) user. Using tactile outputs toprovide haptic feedback to a user enhances the operability of the deviceand makes the user-device interface more efficient (e.g., by helping theuser to provide proper inputs and reducing user mistakes whenoperating/interacting with the device) which, additionally, reducespower usage and improves battery life of the device by enabling the userto use the device more quickly and efficiently.

In some embodiments, a tactile output pattern specifies characteristicsof a tactile output, such as the amplitude of the tactile output, theshape of a movement waveform of the tactile output, the frequency of thetactile output, and/or the duration of the tactile output.

When tactile outputs with different tactile output patterns aregenerated by a device (e.g., via one or more tactile output generatorsthat move a moveable mass to generate tactile outputs), the tactileoutputs may invoke different haptic sensations in a user holding ortouching the device. While the sensation of the user is based on theuser's perception of the tactile output, most users will be able toidentify changes in waveform, frequency, and amplitude of tactileoutputs generated by the device. Thus, the waveform, frequency andamplitude can be adjusted to indicate to the user that differentoperations have been performed. As such, tactile outputs with tactileoutput patterns that are designed, selected, and/or engineered tosimulate characteristics (e.g., size, material, weight, stiffness,smoothness, etc.); behaviors (e.g., oscillation, displacement,acceleration, rotation, expansion, etc.); and/or interactions (e.g.,collision, adhesion, repulsion, attraction, friction, etc.) of objectsin a given environment (e.g., a user interface that includes graphicalfeatures and objects, a simulated physical environment with virtualboundaries and virtual objects, a real physical environment withphysical boundaries and physical objects, and/or a combination of any ofthe above) will, in some circumstances, provide helpful feedback tousers that reduces input errors and increases the efficiency of theuser's operation of the device. Additionally, tactile outputs are,optionally, generated to correspond to feedback that is unrelated to asimulated physical characteristic, such as an input threshold or aselection of an object. Such tactile outputs will, in somecircumstances, provide helpful feedback to users that reduces inputerrors and increases the efficiency of the user's operation of thedevice.

In some embodiments, a tactile output with a suitable tactile outputpattern serves as a cue for the occurrence of an event of interest in auser interface or behind the scenes in a device. Examples of the eventsof interest include activation of an affordance (e.g., a real or virtualbutton, or toggle switch) provided on the device or in a user interface,success or failure of a requested operation, reaching or crossing aboundary in a user interface, entry into a new state, switching of inputfocus between objects, activation of a new mode, reaching or crossing aninput threshold, detection or recognition of a type of input or gesture,etc. In some embodiments, tactile outputs are provided to serve as awarning or an alert for an impending event or outcome that would occurunless a redirection or interruption input is timely detected. Tactileoutputs are also used in other contexts to enrich the user experience,improve the accessibility of the device to users with visual or motordifficulties or other accessibility needs, and/or improve efficiency andfunctionality of the user interface and/or the device. Tactile outputsare optionally accompanied with audio outputs and/or visible userinterface changes, which further enhance a user's experience when theuser interacts with a user interface and/or the device, and facilitatebetter conveyance of information regarding the state of the userinterface and/or the device, and which reduce input errors and increasethe efficiency of the user's operation of the device.

FIGS. 4F-4H provide a set of sample tactile output patterns that may beused, either individually or in combination, either as is or through oneor more transformations (e.g., modulation, amplification, truncation,etc.), to create suitable haptic feedback in various scenarios and forvarious purposes, such as those mentioned above and those described withrespect to the user interfaces and methods discussed herein. Thisexample of a palette of tactile outputs shows how a set of threewaveforms and eight frequencies can be used to produce an array oftactile output patterns. In addition to the tactile output patternsshown in these figures, each of these tactile output patterns isoptionally adjusted in amplitude by changing a gain value for thetactile output pattern, as shown, for example for FullTap 80 Hz, FullTap200 Hz, MiniTap 80 Hz, MiniTap 200 Hz, MicroTap 80 Hz, and MicroTap 200Hz in FIGS. 4I-4K, which are each shown with variants having a gain of1.0, 0.75, 0.5, and 0.25. As shown in FIGS. 4I-4K, changing the gain ofa tactile output pattern changes the amplitude of the pattern withoutchanging the frequency of the pattern or changing the shape of thewaveform. In some embodiments, changing the frequency of a tactileoutput pattern also results in a lower amplitude as some tactile outputgenerators are limited by how much force can be applied to the moveablemass and thus higher frequency movements of the mass are constrained tolower amplitudes to ensure that the acceleration needed to create thewaveform does not require force outside of an operational force range ofthe tactile output generator (e.g., the peak amplitudes of the FullTapat 230 Hz, 270 Hz, and 300 Hz are lower than the amplitudes of theFullTap at 80 Hz, 100 Hz, 125 Hz, and 200 Hz).

FIGS. 4F-4K show tactile output patterns that have a particularwaveform. The waveform of a tactile output pattern represents thepattern of physical displacements relative to a neutral position (e.g.,xzero) versus time that a moveable mass goes through to generate atactile output with that tactile output pattern. For example, a firstset of tactile output patterns shown in FIG. 4F (e.g., tactile outputpatterns of a “FullTap”) each have a waveform that includes anoscillation with two complete cycles (e.g., an oscillation that startsand ends in a neutral position and crosses the neutral position threetimes). A second set of tactile output patterns shown in FIG. 4G (e.g.,tactile output patterns of a “MiniTap”) each have a waveform thatincludes an oscillation that includes one complete cycle (e.g., anoscillation that starts and ends in a neutral position and crosses theneutral position one time). A third set of tactile output patterns shownin FIG. 4H (e.g., tactile output patterns of a “MicroTap”) each have awaveform that includes an oscillation that include one half of acomplete cycle (e.g., an oscillation that starts and ends in a neutralposition and does not cross the neutral position). The waveform of atactile output pattern also includes a start buffer and an end bufferthat represent the gradual speeding up and slowing down of the moveablemass at the start and at the end of the tactile output. The examplewaveforms shown in FIGS. 4F-4K include xmin and xmax values whichrepresent the maximum and minimum extent of movement of the moveablemass. For larger electronic devices with larger moveable masses, theremay be larger or smaller minimum and maximum extents of movement of themass. The examples shown in FIGS. 4F-4K describe movement of a mass in 1dimension, however similar principles would also apply to movement of amoveable mass in two or three dimensions.

As shown in FIGS. 4F-4K, each tactile output pattern also has acorresponding characteristic frequency that affects the “pitch” of ahaptic sensation that is felt by a user from a tactile output with thatcharacteristic frequency. For a continuous tactile output, thecharacteristic frequency represents the number of cycles that arecompleted within a given period of time (e.g., cycles per second) by themoveable mass of the tactile output generator. For a discrete tactileoutput, a discrete output signal (e.g., with 0.5, 1, or 2 cycles) isgenerated, and the characteristic frequency value specifies how fast themoveable mass needs to move to generate a tactile output with thatcharacteristic frequency. As shown in FIGS. 4F-4H, for each type oftactile output (e.g., as defined by a respective waveform, such asFullTap, MiniTap, or MicroTap), a higher frequency value corresponds tofaster movement(s) by the moveable mass, and hence, in general, ashorter time to complete the tactile output (e.g., including the time tocomplete the required number of cycle(s) for the discrete tactileoutput, plus a start and an end buffer time). For example, a FullTapwith a characteristic frequency of 80 Hz takes longer to complete thanFullTap with a characteristic frequency of 100 Hz (e.g., 35.4 ms vs.28.3 ms in FIG. 4F). In addition, for a given frequency, a tactileoutput with more cycles in its waveform at a respective frequency takeslonger to complete than a tactile output with fewer cycles its waveformat the same respective frequency. For example, a FullTap at 150 Hz takeslonger to complete than a MiniTap at 150 Hz (e.g., 19.4 ms vs. 12.8 ms),and a MiniTap at 150 Hz takes longer to complete than a MicroTap at 150Hz (e.g., 12.8 ms vs. 9.4 ms). However, for tactile output patterns withdifferent frequencies this rule may not apply (e.g., tactile outputswith more cycles but a higher frequency may take a shorter amount oftime to complete than tactile outputs with fewer cycles but a lowerfrequency, and vice versa). For example, at 300 Hz, a FullTap takes aslong as a MiniTap (e.g., 9.9 ms).

As shown in FIGS. 4F-4K, a tactile output pattern also has acharacteristic amplitude that affects the amount of energy that iscontained in a tactile signal, or a “strength” of a haptic sensationthat may be felt by a user through a tactile output with thatcharacteristic amplitude. In some embodiments, the characteristicamplitude of a tactile output pattern refers to an absolute ornormalized value that represents the maximum displacement of themoveable mass from a neutral position when generating the tactileoutput. In some embodiments, the characteristic amplitude of a tactileoutput pattern is adjustable, e.g., by a fixed or dynamically determinedgain factor (e.g., a value between 0 and 1), in accordance with variousconditions (e.g., customized based on user interface contexts andbehaviors) and/or preconfigured metrics (e.g., input-based metrics,and/or user-interface-based metrics). In some embodiments, aninput-based metric (e.g., an intensity-change metric or an input-speedmetric) measures a characteristic of an input (e.g., a rate of change ofa characteristic intensity of a contact in a press input or a rate ofmovement of the contact across a touch-sensitive surface) during theinput that triggers generation of a tactile output. In some embodiments,a user-interface-based metric (e.g., a speed-across-boundary metric)measures a characteristic of a user interface element (e.g., a speed ofmovement of the element across a hidden or visible boundary in a userinterface) during the user interface change that triggers generation ofthe tactile output. In some embodiments, the characteristic amplitude ofa tactile output pattern may be modulated by an “envelope” and the peaksof adjacent cycles may have different amplitudes, where one of thewaveforms shown above is further modified by multiplication by anenvelope parameter that changes over time (e.g., from 0 to 1) togradually adjust amplitude of portions of the tactile output over timeas the tactile output is being generated.

Although only specific frequencies, amplitudes, and waveforms arerepresented in the sample tactile output patterns in FIGS. 4F-4K forillustrative purposes, tactile output patterns with other frequencies,amplitudes, and waveforms may be used for similar purposes. For example,waveforms that have between 0.5 to 4 cycles can be used. Otherfrequencies in the range of 60 Hz-400 Hz may be used as well.

User Interfaces and Associated Processes

Attention is now directed towards embodiments of user interfaces (“UI”)and associated processes that may be implemented on an electronicdevice, such as portable multifunction device 100 or device 300, with adisplay generation component, one or more input devices, and(optionally) one or cameras.

FIGS. 5A-5AK illustrate example user interfaces for displaying a visualindication of one or more inputs that if performed would cause criteriato be satisfied for performing an action (e.g., displaying an animationsequence associated with a virtual object), in accordance with someembodiments. For example, in response to detecting an input that doesnot satisfy criteria for triggering an animation associated with avirtual object, a visual indication of input that would satisfy criteriafor triggering the animation is displayed. The user interfaces in thesefigures are used to illustrate the processes described below, includingthe processes in FIGS. 9A-9C, 10A-10C, 11A-11D, 12A-12B, and 13A-13B.For convenience of explanation, some of the embodiments will bediscussed with reference to operations performed on a device with atouch-sensitive display system 112. In such embodiments, the focusselector is, optionally: a respective finger or stylus contact, arepresentative point corresponding to a finger or stylus contact (e.g.,a centroid of a respective contact or a point associated with arespective contact), or a centroid of two or more contacts detected onthe touch-sensitive display system 112. However, analogous operationsare, optionally, performed on a device with a display 450 and a separatetouch-sensitive surface 451 in response to detecting the contacts on thetouch-sensitive surface 451 while displaying the user interfaces shownin the figures on the display 450, along with a focus selector.

In FIG. 5A, a virtual object 5002 (a virtual box) is displayed in anobject staging user interface 5004. In some embodiments, while objectstaging user interface 5004 is displayed, user input for manipulation ofvirtual object 5002 (e.g., an input by a contact at a locationcorresponding to virtual object 5002 displayed on touch-sensitivedisplay system 112) causes a characteristic of virtual object 5002 tochange (e.g. rotation of the virtual object about one or more axes). Inthis way, a user is enabled to view virtual object 5002 from variousangles. In some embodiments, a change made to a characteristic ofvirtual object 5002 while the virtual object is displayed in objectstaging user interface 5004 (e.g., a rotation applied to the virtualobject 5002 about a first axis) is applied to the virtual object 5002when the virtual object 5002 is displayed in a user interface thatincludes at least a portion of a field of view of one or more cameras ofdevice 100 (e.g., an augmented reality view). In this way, a user isenabled to set an orientation that the virtual object 5002 will havewhen displayed in an augmented reality view.

Object staging user interface 5004 includes toggle control 5008 thatindicates a current display mode (e.g., the current display mode is anobject staging user interface mode, as indicated by the highlighted“Object” indicator) and that, when activated, causes transition to aselected display mode. For example, while the object staging userinterface 5004 is displayed, a tap input by a contact at a location thatcorresponds to toggle 5008 (e.g., a location that corresponds to aportion of toggle control 5008 that includes the text “AR”) causes theobject staging user interface 5008 to be replaced by at least a portionof a field of view of one or more cameras of device 100. Object staginguser interface 5004 also includes cancel control 5006 (e.g., forreplacing display of object staging user interface 5004 with a differentuser interface such as a previously displayed user interface) and sharecontrol 5010 (e.g., for displaying a sharing interface).

FIGS. 5B-5C illustrate an input that causes rotation of virtual object5002 while object staging user interface 5004 is displayed. In FIG. 5B,an input by contact 5012 (e.g., a rightward swipe gesture) is detectedat a location that corresponds to virtual object 5002. As the contactmoves along a path indicated by arrow 5014, virtual object 5002 rotatesabout an axis that is perpendicular to the movement of the contact.

In FIG. 5D, an input (e.g., a tap input) by contact 5016 is detected ontouch-sensitive display system 112 at a location that corresponds to the“AR” region of toggle control 5008. In response to the input, display ofobject staging user interface 5004 is replaced by display of virtualobject 5002 in an augmented reality view (e.g., virtual object 5002 isdisplayed in a physical environment captured by one or more cameras ofdevice 100), as illustrated in FIGS. 5E-5F. To indicate that the currentdisplay mode is an augmented reality user interface mode, the “AR”region of toggle control 5008 is highlighted in FIG. 5F.

FIG. 5E illustrates a physical environment 5018 in which device 100 isoperated. Table 5020 is a physical object located in physicalenvironment 5018. The display of device 100 shows virtual object 5002placed on an upper surface of table 5020 in a view of physicalenvironment 5018 as captured by one or more cameras of device 100.

FIG. 5F-5I illustrate an input that causes animation of virtual object5002 to occur. In FIG. 5F, virtual object 5002 is shown in augmentedreality user interface 5022 that includes a view of physical environment5018 as captured by one or more cameras of device 100. In FIG. 5G, aninput (e.g., a tap input) by contact 5024 is detected on touch-sensitivedisplay system 112 at a location that corresponds to virtual object5002. In response to the input, virtual object 5002 is animated (e.g.,an animation sequence shows the lid of the box gradually opening), asshown in FIGS. 5H-5I. In some embodiments, the animation of virtualobject 5002 continues while contact 5024 is maintained at a locationthat corresponds to virtual object 5002. In some embodiments, when theanimation of virtual object 5002 has been initiated, the animationcontinues (e.g., an animation sequence completes and/or an animationsequence continually loops) after contact 5024 lifts off oftouch-sensitive display system 112.

FIGS. 5J-5K illustrate an input that causes movement of virtual object5002 while augmented reality user interface 5022 is displayed. In FIG.5J, an input by contact 5024 (e.g., a rightward swipe gesture) isdetected at a location that corresponds to virtual object 5002. As thecontact moves along a path indicated by arrow 5026, virtual object 5002moves along the surface of table 5020 in the direction of the movementof the contact, as shown in FIG. 5K.

In FIG. 5K, because the input that causes movement of virtual object5002 does not satisfy criteria for causing animation of virtual object5002 to occur, a visual indication 5028 of the input needed to causeanimation to occur is displayed. Visual indication 5028 includes thetext, “Tap on the box to view animation!” In this way, the user isprovided with feedback that provides information about the input thatwill trigger playback of the animation sequence.

FIGS. 5L-50 illustrate a gaze input that causes animation of virtualobject 5002 to occur. In FIG. 5L-1, a user 5030 is gazing at a lowerportion of the display of device 100, as illustrated by gaze indicator5034 that indicates the gaze target of eye 5032 of the user 5030. FIG.5L-2 illustrates augmented reality user interface 5022 displayed bydevice 100. In FIG. 5L-2, gaze 5034 is targeted at a location inaugmented reality user interface 5022 that does not correspond tovirtual object 5002. A visual indication 5036 is displayed, indicatinginput that would cause animation of virtual object 5002 to occur (e.g.,“Look at the box to see animation!”).

In FIG. 5M-1, user 5030 is gazing at an upper portion of the display ofdevice 100, as illustrated by gaze indicator 5038. In FIG. 5M-2, gaze5038 is targeted at a location in augmented reality user interface 5022that corresponds to virtual object 5002. As shown in FIGS. 5N-2 and5O-2, in accordance with a determination that gaze 5038 is targeted atvirtual object 5002, the visual indication 5036 ceases to be displayedand animation of virtual object 5002 occurs. In some embodiments, theanimation of virtual object 5002 continues while gaze 5038 continues totarget virtual object 5002, as illustrated in FIGS. 5N-1 to 5N-2 and5O-1 to 5O-2. In some embodiments, the animation of virtual object 5002halts in accordance with a determination that the gaze is not targetedat virtual object 5002. In some embodiments, when the animation ofvirtual object 5002 has been initiated, the animation continues (e.g.,an animation sequence completes and/or an animation sequence continuallyloops) after the gaze moves away from virtual object 5002.

FIGS. 5P-5R illustrate an input that attempts to place virtual object5002 on a surface with a size that is not compatible with the virtualobject 5002. In FIG. 5P, an input by contact 5040 is detected at alocation that corresponds to virtual object 5002. As contact 5040 movesalong a path indicated by arrow 5042, virtual object 5002 moves alongfloor 5046 (the floor of physical environment 5018 as captured in thefield of view of the one or more cameras of device 100) in the directionof the movement of the contact, as illustrated in FIGS. 5P-5Q. In FIGS.5Q-5R, contact 5040 continues to move along a path indicated by arrow5048 as the user attempts to place virtual object 5002 on the uppersurface of lamp 5044. In FIG. 5R, because the size of the upper surfaceof lamp 5044 is not compatible with the size of virtual object 5002(e.g., including a size corresponding to an animation sequenceassociated with virtual object 5002), a visual indication 5050 isdisplayed, indicating that the size of the upper surface of lamp 5044 isnot compatible with virtual object 5002 (e.g., “The object cannot beplaced on this surface!”).

FIGS. 5S-5W illustrate positioning of virtual object 5002 in thedisplayed field of view of cameras of device 100 that causes animationof virtual object 5002 to occur. In FIG. 5S, virtual object 5002 hasbeen placed on the upper surface of physical table 5020 (e.g., such thatvirtual object 5002 is displayed at a fixed position relative tophysical table 5020 as the field of view of the cameras of device 100changes). Virtual object 5002 and physical table 5020 are only partiallyin the displayed field of view of the cameras of device 100. Because thevirtual object 5002 is not fully displayed in the field of view of thecameras of device 100, a visual indication 5054 is displayed, indicatinginput that would cause animation of virtual object 5002 to occur (e.g.,“Move the box to the center to view animation!”). Circle 5052 indicatesthe location to which the virtual object is to be moved to causeanimation of virtual object 5002 to occur. In some embodiments, circle5052 is not displayed by device 100 (as indicated by the dotted lines).In some embodiments, circle 5052 is a prompt that is displayed by device100. In some embodiments, animation of virtual object 5002 occurs whenvirtual object 5002 is fully displayed in the displayed portion of thefield of view of the cameras of device 100.

In FIG. 5T, device 100 has been moved relative to physical environment5018 such that virtual object 5002 is partially displayed within circle5052. Because virtual object 5002 is not fully displayed within circle5052, animation of virtual object 5002 has not occurred and visualindication 5054 continues to be displayed. In FIG. 5U, device 100 hasbeen moved relative to physical environment 5018 such that virtualobject 5002 is fully displayed within circle 5052. Because virtualobject 5002 is fully displayed within circle 5052, animation of virtualobject 5002 occurs (as illustrated in FIGS. 5U, 5V, and 5W) and visualindication 5054 ceases to be displayed.

FIGS. 5X-5AC illustrate an embodiment in which multiple criteria must besatisfied to cause playback of an animation sequence of virtual object5002. In FIG. 5X, virtual object 5002 and physical table 5020 are onlypartially in the displayed field of view of the cameras of device 100.Until both criteria are satisfied, a visual indication 5056 isdisplayed, indicating input that would cause animation of virtual object5002 to occur (e.g., “Move the box to the center and tap on the box toview animation!”). Circle 5052 indicates the location to which thevirtual object is to be moved to cause animation of virtual object 5002to occur.

In FIG. 5Y, device 100 has been moved relative to physical environment5018 such that virtual object 5002 is partially displayed within circle5052. Because the multiple criteria have not been satisfied, animationof virtual object 5002 has not occurred and visual indication 5054continues to be displayed. In FIG. 5Z, device 100 has been movedrelative to physical environment 5018 such that virtual object 5002 isfully displayed within circle 5052, satisfying a first criterion ofmultiple criteria that must be satisfied to cause playback of theanimation sequence of virtual object 5002. Because the first criterionhas been satisfied but the second criterion has not been satisfied,visual indication 5056 is updated to indicate the input that must beprovided to satisfy the second criterion (e.g., “Tap on the box to viewanimation!”). In FIG. 5AA, an input (e.g., a tap input) by a contact5058 is detected at a location that corresponds to virtual object 5002.In response to the input, because the first criterion and the secondcriterion have been met, animation of virtual object 5002 occurs (asillustrated in FIGS. 5AA, 5AB, and SAC) and visual indication 5054ceases to be displayed.

FIGS. 5AD-5AH illustrate movement of device 100 that causes playback ofan animation of virtual object 5002 to occur. In FIG. 5AD, a distancebetween a displayed position of virtual object 5002 and device 100 isgreater than a threshold distance for an animation of virtual object5002 to occur. A visual indication 5060 is displayed to indicate inputthat would cause animation of virtual object 5002 to occur (e.g., “Movecloser to the box to view animation!”). In FIG. 5AE, device 100 hasmoved closer to the displayed position of virtual object 5002. Becausethe distance between the displayed position of virtual object 5002 anddevice 100 is still greater than the threshold distance for an animationof virtual object 5002 to occur, animation of virtual object 5002 hasnot occurred and visual indication 5060 continues to be displayed. InFIG. 5AF, device 100 has moved closer to the displayed position ofvirtual object 5002 such that the distance between the displayedposition of virtual object 5002 and device 100 has decreased below thethreshold distance for an animation of virtual object 5002 to occur.Because the distance between the displayed position of virtual object5002 and device 100 has decreased below the threshold distance,animation of virtual object 5002 occurs (as illustrated in FIGS. 5AF,5AG, and 5AH) and visual indication 5060 ceases to be displayed.

FIGS. 5AI-5AK illustrate a play head control for adjusting playback ofan animated sequence that animates virtual object 5002. In FIG. 5AI, atimeline 5062 is displayed that corresponds to playback of the animationsequence for virtual object 5002. Play head 5064 indicates a positionwithin timeline 5062 of a currently displayed frame of the animationsequence. An input by contact 5066 is detected at a location thatcorresponds to play head 5064. As contact 5066 moves along a pathindicated by arrow 5068, play head 5064 moves along timeline 5062 in thedirection of the movement of the contact (e.g., play head 5064 is“dragged” by the movement of contact 5066) and the animation sequence ofvirtual object 5002 progresses in accordance with the movement of playhead 5064, as illustrated in FIGS. 5AI-5AJ. In FIGS. 5AJ-5AK, contact5066 continues to move along a path indicated by arrow 5070, causingfurther movement of play head 5064 and further progression of theanimation sequence of virtual object 5002.

FIGS. 6A-6AI illustrate example user interfaces for displaying a mediaitem in a user interface that includes content of at least a portion ofa field of view of one or more cameras (e.g., displaying a photograph inan augmented reality environment), in accordance with some embodiments.The user interfaces in these figures are used to illustrate theprocesses described below, including the processes in FIGS. 9A-9C,10A-10C, 11A-11D, 12A-12B, and 13A-13B. For convenience of explanation,some of the embodiments will be discussed with reference to operationsperformed on a device with a touch-sensitive display system 112. In suchembodiments, the focus selector is, optionally: a respective finger orstylus contact, a representative point corresponding to a finger orstylus contact (e.g., a centroid of a respective contact or a pointassociated with a respective contact), or a centroid of two or morecontacts detected on the touch-sensitive display system 112. However,analogous operations are, optionally, performed on a device with adisplay 450 and a separate touch-sensitive surface 451 in response todetecting the contacts on the touch-sensitive surface 451 whiledisplaying the user interfaces shown in the figures on the display 450,along with a focus selector.

In FIG. 6A, a currently selected photo 6002 is displayed in a photoviewing interface 6004 of a photo management application. Photomanagement application user interface includes a photo thumbnailnavigation bar 6006 that displays thumbnails of photos (e.g., photoscaptured by device 100). In some embodiments, leftward or rightwardswipe input on photo thumbnail navigation bar 6006 causes scrolling ofthe displayed set of photo thumbnails in the direction of the swipeinput, such that additional photo thumbnails are revealed. In someembodiments, an input (e.g., a tap input) on a photo thumbnail displayedin photo thumbnail navigation bar 6006 causes an enlarged version of aphoto to be selected for display in photo viewing interface 6004. Forexample, because photo 6002 is currently selected (e.g., as indicated byframe 6003 surrounding the photo thumbnail that corresponds to currentlyselected photo 6002) an enlarged version of photo 6002 is displayed inphoto viewing interface 6004. Photo viewing interface 6004 includescontrol 6008 for displaying a sharing user interface, control 6010 forapplying metadata to a selected photo (e.g., metadata indicating thatthe selected photo is a favorite photo), and control 6012 for deleting aselected photo.

In FIG. 6B, an input (e.g., a tap input) by contact 6014 is detected ata location that corresponds to control 6008 for displaying a sharinguser interface. In response to the input, sharing user interface 6016 isdisplayed overlaying photo viewing interface 6004, as illustrated inFIG. 6C. Sharing user interface 6016 includes an indication 6018 ofcontacts with whom currently selected photo 6002 may be shared, anindication 6020 of communication modes via which currently selectedphoto 6002 may be shared, and additional functions 6022 that may beapplied to currently selected photo 6002. While sharing user interface6016 is displayed, control 6028 for ceasing to display the sharing userinterface 6016 is displayed.

In FIG. 6C, an input (e.g., a tap input) by contact 6026 is detected ata location that corresponds to control 6024 for displaying one or moreselected objects in an object three-dimensional viewing mode (e.g.,object management user interface 5004 or augmented reality userinterface 5022). In response to the input, sharing user interface 6016and photo viewing interface 6004 cease to be displayed and currentlyselected photo 6002 is displayed with a field of view of one or morecameras of device 100, as illustrated in FIGS. 6D-6E.

FIG. 6D illustrates a physical environment 5018 in which device 100 isoperated. Physical table 5020 appears in the physical environment 5018and in a view of physical environment 5018 captured by one or morecameras of device 100, as displayed by device 100. Photo 6002 asdisplayed by device 100 appears to be hanging on wall 6032 of thephysical environment 5018.

FIG. 6E-6G illustrate adjustment to the displayed portion of the fieldof view of cameras of the device 100 that occurs as device 100 moves inthe physical environment. In FIG. 6E, photo 6002 is shown in augmentedreality user interface 5022 that includes a view of the physicalenvironment. In some embodiments, when displayed in augmented realityuser interface 5022, photo 6002 appears to have a three-dimensionalappearance (e.g., a simulated backing 6034 of photo 6002 creates a spacebetween wall 6032 and a displayed position of photo 6002). From FIG. 6Eto FIG. 6F, as device 100 moves from a first position in the physicalenvironment to a second position in the physical environment, thedisplayed locations of photo 6002 and table 5020 change, indicating thatphoto 6002 is displayed at a fixed position relative to the physicalenvironment. From FIG. 6F to FIG. 6G, device 100 continues to move fromthe second position in the physical environment to a third position inthe physical environment, and the displayed locations of photo 6002 andtable 5020 continue to change in accordance with the movement of device100.

FIGS. 6H-6L illustrate successive selection of multiple photographs.

In FIG. 6H, multiple photos are displayed in a photo selection interface6036 of photo management application. An input (e.g., a tap input) bycontact 6040 is detected on touch-sensitive display system 112 at alocation that corresponds to a “Select” control for enabling a selectionmode in which input received at respective photos will cause selectionof the respective photos. In response to the input, the selection modeis enabled.

In FIG. 6I, an input (e.g., a tap input) by contact 6042 is detected ata location that corresponds to photo 6002. In response to the input,photo 6002 is selected (e.g., as marked by selection indicator 6044displayed adjacent to photo 6002).

In FIG. 6J, an input (e.g., a tap input) by contact 6048 is detected ata location that corresponds to photo 6046. In response to the input,photo 6046 is selected (e.g., as marked by selection indicator 6050displayed adjacent to photo 6046).

In FIG. 6K, an input (e.g., a tap input) by contact 6054 is detected ata location that corresponds to photo 6052. In response to the input,photo 6052 is selected (e.g., as marked by selection indicator 6056displayed adjacent to photo 6052).

In FIG. 6L, while photos 6002, 6046, and 6052 are selected, an input(e.g., a tap input) by contact 6058 is detected at a location thatcorresponds to share control 6008. In response to the input, sharinguser interface 6016 (described above with regard to FIG. 6C) isdisplayed, as illustrated in FIG. 6M. While sharing user interface 6016is displayed, currently selected photos 6002, 6046, and 6052 aredisplayed (e.g., the set of multiple photos, including the photosdisplayed in photo selection interface 6036, is filtered such that onlycurrently selected photos are displayed while sharing user interface6016 is displayed).

FIGS. 6M-6Z illustrate input for manipulating the multiple photographs.

In FIG. 6M, an input (e.g., a tap input) by contact 6060 is detected ata location that corresponds to control 6024 for displaying one or moreselected photos in an object three-dimensional viewing mode. In responseto the input, sharing user interface 6016 and photo selection interface6036 cease to be displayed and currently selected photos 6002, 6046, and6052 are displayed in augmented reality user interface 5022, asillustrated in FIGS. 6N-6O.

FIG. 6N illustrates a physical environment 5018 in which device 100 isoperated. Photos 6002, 6046, and 6052 as displayed by device 100 appearto be hanging on rear wall 6032 of the physical environment 5018.

In FIG. 6O, an input (e.g., a leftward swipe input) by contact 6062 isdetected at a location that corresponds to photo 6002. As contact 6062moves along a path indicated by arrow 6064, photo 6002 moves along rearwall plane 6032 detected in the physical environment. In FIGS. 6O-6P,photo 6002 moves from rear wall plane 6032 onto side wall plane 6031.

In FIG. 6Q, an input (e.g., a leftward swipe input) by contact 6066 isdetected at a location that corresponds to photo 6046. In FIGS. 6Q-6R,as contact 6066 moves along a path indicated by arrow 6068, photo 6046moves leftward along rear wall plane 6032 in the direction indicated byarrow 6068 to a new position as illustrated in FIG. 6R.

In FIG. 6R, an input (e.g., a leftward swipe input) by contact 6070 isdetected at a location that corresponds to photo 6052. In FIGS. 6R-6S,as contact 6070 moves along a path indicated by arrow 6072, photo 6052moves leftward along rear wall plane 6032 in the direction indicated byarrow 6072 to a new position as illustrated in FIG. 6S.

In FIG. 6S, an input (e.g., a upward swipe input) by contact 6074 isdetected at a location that corresponds to photo 6052. In FIGS. 6S-6T,as contact 6074 moves along a path indicated by arrow 6076, photo 6052moves upward along rear wall plane 6032 in the direction indicated byarrow 6076 to a new position as illustrated in FIG. 6T.

In FIG. 6U, an input (e.g., a pinch gesture) by contacts 6086 and 6088for changing the simulated physical size of photo 6046 is detected. Ascontact 6086 moves along a path indicated by arrow 6090 and contact 6088moves along a path indicated by arrow 6092, the size of photo 6046decreases, as illustrated in FIGS. 6U-6V.

In FIG. 6W, an input by contacts 6086 and 6088 is detected for rotatingphoto 6046. As contact 6086 moves along a path indicated by arrow 6094and contact 6088 moves along a path indicated by arrow 6096, photo 6046rotates about an axis that is perpendicular to rear wall plane 6032, asillustrated in FIGS. 6W-6X. From FIG. 6X-FIG. 6Y, photo 6046 continuesto rotate as contact 6086 moves along a path indicated by arrow 6098 andcontact 6088 moves along a path indicated by arrow 6100.

In FIG. 6Z, an input by contacts 6102 and 6104 is detected for rotatingphoto 6046 about an axis 6016 that is parallel to rear wall plane 6032,as illustrated by rotation arrow 6108. Because rotation about an axisthat is parallel to rear wall plane 6032 is not permitted, a visualindication 6098 of the non-permitted activity is displayed (e.g.,including text indicating, “Cannot rotate in this direction!”).

In FIG. 6AA, a currently selected photo 6110 is displayed in photoviewing interface 6004 of the photo management application (e.g., asindicated by frame 6003 surrounding the photo thumbnail that correspondsto currently selected photo 6110). An input (e.g., a tap input) bycontact 6112 is detected at a location that corresponds to control 6008for displaying a sharing user interface 6016. In response to the input,sharing user interface 6016 is displayed overlaying photo viewinginterface 6004, as illustrated in FIG. 6AB.

In FIG. 6AB, an input (e.g., a long press or deep press input) bycontact 6114 is detected at a location that corresponds to control 6024for displaying one or more selected objects in an objectthree-dimensional viewing mode (e.g., object management user interface5004 or augmented reality user interface 5022). In response to theinput, a menu 6118 is displayed for indicating a display mode fordisplaying one or more currently selected photos in the objectthree-dimensional viewing mode (e.g., an “Object” mode or a “Cutout”mode), as illustrated in FIG. 6AC.

In FIG. 6AC, an input (e.g., a tap input) by contact 6116 is detected ata location that corresponds to the “Cutout” region of menu 6118. Inresponse to the input, sharing user interface 6016 and photo viewinginterface 6004 cease to be displayed and a cutout version of image 6110is displayed with a field of view of one or more cameras of device 100,as illustrated in FIGS. 6AD-6AE.

FIG. 6AD illustrates a physical environment 5018 in which device 100 isoperated. Physical table 5020 appears in the physical environment 5018and in a view of physical environment 5018 captured by one or morecameras of device 100, as displayed by device 100. The cutout version ofphoto 6110 as displayed by device 100 appears to be in front of table5020 in the physical environment 5018.

FIG. 6AE-6AI illustrate adjustment to the displayed portion of the fieldof view of cameras of device 100 that occurs as the device moves in thephysical environment. In FIG. 6AE, the cutout version of photo 6110 isshown in augmented reality user interface 5022 that includes a view ofthe physical environment. From FIG. 6AE to FIG. 6AF and from FIG. 6AF toFIG. 6AG, device 100 moves from a position in front of the cutoutversion of image 6110 to a position to the side of cutout version ofimage 6110, revealing that the cutout version of image 6110 has asimulated backing 6120 (e.g., with a simulated thickness that is smallrelative to the size of the cutout version of image 6110, such as asimulated thickness of 1 cm-10 cm). In some embodiments, the cutoutversion of image 6110 displayed in augmented reality user interface 5022is a two-dimensional image that is displayed without a backing (andwould thus not be visible from a side view as shown in FIG. 6AH). FromFIG. 6AG to FIG. 6AH and from FIG. 6AH to FIG. 6AI, device 100 continuesto move from the position to the side of the cutout version of image6110 to a position behind the cutout version of image 6110, and thedisplayed locations of image 6110 and table 5020 continue to change inaccordance with the movement of device 100. In FIG. 6AI, the cutoutversion of image 6110 appears as a reverse version of the cutout versionof image 6110 as viewed from the position in front of the cutout versionof image 6110 (as shown in FIG. 6AE).

FIGS. 7A-7AQ illustrate example user interfaces for responding to aninput directed to a respective virtual object of a plurality ofdisplayed virtual objects, in accordance with some embodiments. The userinterfaces in these figures are used to illustrate the processesdescribed below, including the processes in FIGS. 9A-9C, 10A-10C,11A-11D, 12A-12B, and 13A-13B. For convenience of explanation, some ofthe embodiments will be discussed with reference to operations performedon a device with a touch-sensitive display system 112. In suchembodiments, the focus selector is, optionally: a respective finger orstylus contact, a representative point corresponding to a finger orstylus contact (e.g., a centroid of a respective contact or a pointassociated with a respective contact), or a centroid of two or morecontacts detected on the touch-sensitive display system 112. However,analogous operations are, optionally, performed on a device with adisplay 450 and a separate touch-sensitive surface 451 in response todetecting the contacts on the touch-sensitive surface 451 whiledisplaying the user interfaces shown in the figures on the display 450,along with a focus selector.

In FIG. 7A, virtual object 7002 (a virtual refrigerator), virtual object7004 (a virtual washing machine), and virtual object 7006 (a virtualtelevision) are displayed in object staging user interface 5004 (asdescribed further with regard to FIG. 5A).

FIGS. 7B-7C illustrate rotation of virtual objects 7002, 7004 and 7006in response to an input detected while object staging user interface5004 is displayed.

In FIG. 7B, an input (e.g., an upward vertical swipe input) by contact7008 is detected at a location that corresponds to virtual object 7002.As contact 7008 moves along a path indicated by arrow 7010, virtualobjects 7002, 7004 and 7006 rotate in the direction indicated by arrow7010 (e.g., tilting upward). A virtual boundary 7012 (not displayed, asindicated by the dotted lines) surrounds virtual objects 7002, 7004 and7006. As illustrated in FIG. 7C, in response to the vertical swipe inputby contact 7008, virtual objects 7002, 7004 and 7006 rotate about ahorizontal centerline 7014 of virtual boundary 7012. For example,horizontal centerline 7014 is a centerline that is perpendicular to thedirection of arrow 7016 indicating the direction of the detected upwardvertical swipe input. In some embodiments, in response to a leftward orrightward horizontal swipe input, virtual objects 7002, 7004 and 7006rotate collectively in the direction of the horizontal swipe input abouta vertical centerline (not shown) that bisects boundary 7012. In someembodiments, in response to a leftward or rightward horizontal swipeinput, virtual objects 7002, 7004 and 7006 each rotate in the directionof the horizontal swipe input about vertical centerlines that bisecteach virtual object. In FIG. 7D, contact 7008 has lifted off oftouch-sensitive di splay system 112.

In FIG. 7E, an input (e.g., a tap input) by contact 7018 is detected ata location that corresponds to the “AR” region of toggle control 5008.In response to the input, display of object staging user interface 5004is replaced by display of virtual objects 7002, 7004 and 7006 in anaugmented reality view (e.g., virtual objects 7002, 7004 and 7006 aredisplayed in a physical environment captured by one or more cameras ofdevice 100), as illustrated in FIGS. 7F-7G. In some embodiments, arotation applied to virtual objects 7002, 7004 and 7006 (e.g., rotationabout horizontal centerline 7014, as described with regard to FIGS.7B-7C) in object staging user interface 5004 is not maintained when thevirtual objects are displayed in augmented reality user interface 5022.For example, virtual objects 7002, 7004 and 7006 are configured to beplaced at a fixed position relative to a horizontal plane, and are thusplaced with a horizontal surface substantially parallel to a detectedhorizontal plane. In some embodiments, a rotation applied to virtualobjects 7002, 7004 and 7006 (e.g., rotation about a vertical centerline)in object staging user interface 5004 is maintained when the virtualobjects are displayed in augmented reality user interface 5022.

As illustrated in FIGS. 7F-7G, when a field of view of the cameras isinitially displayed, translucent representations of the virtual objectsmay be displayed (e.g., while one or more planes that correspond to oneor more of the virtual objects have not been detected in a field of viewof the cameras of device 100). In some embodiments, in response todetecting that the object-placement criteria are met (e.g., one or moreplanes that correspond to the one or more virtual objects have beendetected in the field of view of the cameras), the device displays ananimated transition showing one or more of the virtual objects moving(e.g., rotating, scaling, translating, and/or a combination of theabove) from a first orientation (e.g., as shown in FIG. 7G) to thesecond orientation (e.g., as shown in FIG. 7H) and changing from havinga first set of visual properties (e.g., virtual objects 7002, 7004 and7006 are shown in a translucent state in FIG. 7G) to having a second setof visual properties (e.g., virtual objects 7002, 7004 and 7006 areshown in a non-translucent state in FIG. 7H). For example, once one ormore planes for placing virtual objects 7002, 7004 and 7006 areidentified in the camera's field of view, the virtual objects 7002, 7004and 7006 are placed onto that plane with the visible adjustment of itsorientation, size, and translucency (and the like).

In FIGS. 7F-7G, translucent representations of virtual objects 7002,7004 and 7006 are displayed in augmented reality user interface 5022that includes a view of physical environment as captured by one or morecameras of device 100. The translucent representations of virtualobjects 7002, 7004 and 7006 are displayed at fixed positions relative totouch-sensitive display system 112. For example, from FIG. 7F to FIG.7G, as device 100 is moved relative to the physical environment (asindicated by, e.g., the changed position of physical table 7020 in thefield of view of the cameras), virtual objects 7002, 7004 and 7006remain at fixed positions relative to touch-sensitive display system112.

In 7H, a plane 7022 that corresponds to virtual objects 7002, 7004 and7006 (e.g., a horizontal plane that corresponds to a floor in thephysical environment) has been detected in the field of view of thecameras and virtual objects 7002, 7004 and 7006 are placed on thedetected plane in augmented reality user interface 5022. When virtualobjects 7002, 7004 and 7006 have been placed on the detected plane, thevirtual objects remain at a fixed position relative to the detectedplane as device 100 moves in the physical environment and the field ofview of the one or more cameras changes (as described below with regardto FIGS. 7R-7S).

FIGS. 7H-7R illustrate inputs that alter positions and orientations ofvirtual objects 7002, 7004 and 7006 displayed in augmented reality userinterface 5022. In FIG. 7H, an input (e.g., an upward vertical swipeinput) by contact 7022 is detected at a location that corresponds tovirtual object 7002. As contact 7024 moves along a path indicated byarrow 7026, virtual object 7002 moves along plane 7022 in the directionindicated by arrow 7026, as illustrated in FIGS. 7H-7I. As virtualobject 7002 moves along plane 7022, a size of virtual object 7002 isadjusted (e.g., based on a virtual distance from the representation ofthe virtual object to the user, to maintain an accurate perspective ofthe virtual object in the field of view).

In FIG. 7I, as contact 7024 continues to move along a path indicated byarrow 7028, virtual object 7002 moves along plane 7022 in the directionindicated by arrow 7028, as illustrated in FIGS. 7I-7J. As virtualobject 7002 moves along plane 7022 toward the user, the size of virtualobject 7002 increases.

In FIG. 7J, an input (e.g., a de-pinch gesture) by contacts 7030 and7032 for changing the simulated physical size of virtual object 7002 isdetected. As contact 7030 moves along a path indicated by arrow 7034 andcontact 7032 moves along a path indicated by arrow 7036, the size ofvirtual object 7002 increases, as illustrated in FIGS. 7J-7K (e.g.,based on a virtual distance from the representation of the virtualobject to the user, to maintain an accurate perspective of the virtualobject in the field of view).

In FIG. 7K, an input (e.g., a downward and leftward swipe input) bycontact 7038 is detected at a location that corresponds to virtualobject 7006. As contact 7038 moves along a path indicated by arrow 7040,virtual object 7006 moves along plane 7022 in the direction indicated byarrow 7040, as illustrated in FIGS. 7K-7L. As virtual object 7006 movesalong plane 7022 toward the user, the size of virtual object 7006increases.

In FIG. 7L, an input (e.g., a downward and rightward swipe input) bycontact 7042 is detected at a location that corresponds to virtualobject 7004. As contact 7042 moves along a path indicated by arrow 7044,virtual object 7004 moves along plane 7022 in the direction indicated byarrow 7044, as illustrated in FIGS. 7L-7M. As virtual object 7004 movesalong plane 7022 toward the user, the size of virtual object 7004increases.

In FIG. 7M, an input (e.g., a de-pinch gesture) by contacts 7046 and7048 for changing the simulated physical size of virtual object 7004 isdetected. As contact 7046 moves along a path indicated by arrow 7050 andcontact 7048 moves along a path indicated by arrow 7052, the size ofvirtual object 7004 increases, as illustrated in FIGS. 7M-7N.

In FIG. 7N, an input (e.g., a pinch gesture) by contacts 7054 and 7056for changing the simulated physical size of virtual object 7002 isdetected. As contact 7054 moves along a path indicated by arrow 7058 andcontact 7056 moves along a path indicated by arrow 7060, the size ofvirtual object 7002 decreases, as illustrated in FIGS. 7N-7O.

In FIG. 7O, an input (e.g., a pinch gesture) by contacts 7062 and 7064for changing the simulated physical size of virtual object 7004 isdetected. As contact 7062 moves along a path indicated by arrow 7066 andcontact 7064 moves along a path indicated by arrow 7068, the size ofvirtual object 7002 decreases, as illustrated in FIGS. 7O-7P.

In FIG. 7P, an input (e.g., an upward and leftward swipe input) bycontact 7070 is detected at a location that corresponds to virtualobject 7004. As contact 7070 moves along a path indicated by arrow 7072,virtual object 7004 moves along plane 7022 in the direction indicated byarrow 7072, as illustrated in FIGS. 7P-7Q. As virtual object 7004 movesalong plane 7022 away from the user, the size of virtual object 7004decreases (e.g., based on a virtual distance from the representation ofthe virtual object to the user, to maintain an accurate perspective ofthe virtual object in the field of view).

In FIG. 7Q, an input (e.g., a rotation gesture) by contacts 7074 and7076 for changing the simulated orientation of virtual object 7006 isdetected. As contact 7074 moves along a path indicated by arrow 7078 andcontact 7076 moves along a path indicated by arrow 7080, virtual object7006 rotates about an axis that is perpendicular to plane 7022 relativeto which virtual object 7006 is positioned, as illustrated in FIGS. 7Q-R

From FIG. 7R to FIG. 7S and from FIG. 7S to FIG. 7T, as device 100 ismoved relative to the physical environment (as indicated by, e.g., thechanged position of physical table 7020 in the field of view of thecameras), virtual objects 7002, 7004 and 7006 remain at fixed positionsrelative to plane 7022 in the physical environment (e.g., the positionsof virtual objects 7002, 7004, and 7006 move with the physicalenvironment as the view of the physical environment changes).

In FIG. 7T, an input (e.g., a tap input) by contact 7082 is detected inthe augmented reality user interface 5022. In response to the input,toggle control 5008, cancel control 5006, and share control 5010 aredisplayed, as shown in FIG. 7U.

In FIG. 7U, an input (e.g., a tap input) by contact 7084 is detected ata location that corresponds to the “Object” region of toggle control5008. In response to the input, display of virtual objects 7002, 7004and 7006 in augmented reality user interface 5022 is replaced by displayof virtual objects 7002, 7004 and 7006 in object staging view 5004, asillustrated in FIGS. 7U, 7V, 7W, and 7X. As the transition fromdisplaying virtual objects 7002, 7004 and 7006 in augmented reality userinterface 5022 to displaying virtual objects 7002, 7004 and 7006 inobject staging view 5004 occurs, the device displays an animatedtransition showing virtual objects 7002, 7004 and 7006 moving (e.g.,rotating, scaling, translating, and/or a combination of the above) backto a predefined configuration (e.g., predefined order, positions,orientations, and/or sizes) of the virtual objects 7002, 7004 and 7006(as previously displayed in object user interface 5004 in FIG. 7A). Forexample, from FIG. 7U to FIG. 7X: virtual object 7002 moves downward andto the left to the position in which virtual object 7002 was previouslydisplayed in FIG. 7A; virtual object 7004 moves downward and to theright to the position in which virtual object 7004 was previouslydisplayed in FIG. 7A; and virtual object 7006 moves upward, to theright, and rotates counter-clockwise to the position and orientation inwhich virtual object 7006 was previously displayed in FIG. 7A. In someembodiments, while virtual objects 7002, 7004 and 7006 are displayed inobject staging view 5004, the virtual object are displayed ordered byheight (e.g., in decreasing height order) and/or with equal spacingbetween the displayed virtual objects.

In FIG. 7X, an input (e.g., a tap input) by contact 7086 is detected ata location that corresponds to the “Object” region of toggle control5008. In response to the input, display of virtual objects 7002, 7004and 7006 in object staging view 5004 is replaced by display of virtualobjects 7002, 7004 and 7006 in augmented reality user interface 5022, asillustrated in FIGS. 7X-7Y. In FIG. 7Y, virtual objects 7002, 7004 and7006 are displayed in accordance with positions, orientations, and sizesthat resulted from adjustments previously made to of virtual objects7002, 7004 and 7006 as described with regard to FIGS. 7H-7R.

FIGS. 7Z-7AC illustrate boundaries of virtual objects used to setspacing between virtual objects in the object staging user interface5004. In FIG. 7Z, boundary 7094 corresponds to virtual object 7088 andboundary 7098 corresponds to virtual object 7092. Boundary 7096corresponds to a static state of virtual object 7090.

An animation sequence is associated with virtual object 7090. Timeline7100 indicates a playback position within the animation sequenceassociated with virtual object 7090. A play control 7102 for initiatingplayback of the animation sequence associated with virtual object 7090and a volume control 7104 for adjusting a volume of sound associatedwith the animation sequence are displayed. In FIG. 7Z, an input (e.g., atap input) by contact 7106 is detected at a location that corresponds toplay control 7102. In response to the input, the animation sequenceassociated with virtual object 7090 is activated, as illustrated bymovement of the virtual object 7090 in FIGS. 7Z-7AC. The movement ofvirtual object 7090 extends beyond first boundary 7096. Boundary 7108corresponds to the extended range of virtual object 7090 over theduration of the animation sequence. Equal spacing between virtualobjects 7088, 7090, and 7092 in the object staging user interface 5004is set based on boundary 7094, boundary 7108, and boundary 7098, asindicated in FIG. 7AC.

FIGS. 7AD-7AL illustrate selection of multiple virtual objects to placein an augmented reality user interface 5022.

FIG. 7AD illustrates an object management user interface 7099 (e.g., awebsite, a photo management application user interface, and/or a filemanagement user interface) that displays two-dimensional images, such asimage 7118, and/or two-dimensional representations of three-dimensionalvirtual objects, such as virtual object 7110.

In FIG. 7AE, an input (e.g., a tap input) by contact 7112 is detected ata location that corresponds to virtual object 7110. In response to theinput, virtual object 7110 is selected (e.g., as marked by selectionindicator 7114 displayed adjacent to virtual object 7112).

In FIG. 7AF, an input (e.g., a tap input) by contact 7116 is detected ata location that corresponds to image 7118. In response to the input,image 7118 is selected (e.g., as marked by selection indicator 7120displayed overlaying image 7118).

In FIG. 7AG, an input (e.g., a tap input) by contact 7124 is detected ata location that corresponds to virtual object 7122. In response to theinput, virtual object 7122 is selected (e.g., as marked by selectionindicator 7126 displayed adjacent to virtual object 7122).

In FIG. 7AH, while virtual objects 7110, 7118 and 7122 are selected, aninput (e.g., a tap input) by contact 7128 is detected at a location thatcorresponds to share control 7130. In response to the input, sharinguser interface 6016 (described above with regard to FIG. 6C) isdisplayed, as illustrated in FIG. 7AI. While sharing user interface 6016is displayed, currently selected virtual objects 7110, 7118 and 7122 aredisplayed (e.g., a collection of objects and/or images, including theobjects and/or images displayed in object management user interface7099, is filtered such that only currently selected virtual objects aredisplayed while sharing user interface 6016 is displayed).

In FIG. 7AI, an input (e.g., a tap input) by contact 7128 is detected ata location that corresponds to control 6024 for displaying one or moreselected objects in an object three-dimensional viewing mode. Inresponse to the input, sharing user interface 6016 and object managementuser interface 7099 cease to be displayed and currently selected virtualobjects 7110, 7118 and 7122 are displayed in an object staging userinterface 5004, as illustrated in FIG. 7AJ. (In some embodiments, inresponse to the input for displaying the one or more selected objects inthe object three-dimensional viewing mode, the one or more selectedobjects are displayed in an augmented reality user interface 5022, asdescribed with regard to FIG. 7AK.)

In FIG. 7AJ, an input (e.g., a tap input) by contact 7130 is detected ata location that corresponds to the “AR” region of toggle control 5008.In response to the input, display of virtual objects 7110, 7118 and 7122in object staging view 5004 is replaced by display of virtual objects7110, 7118 and 7122 in augmented reality user interface 5022.

In some embodiments, a plane type is defined for a virtual object and,when the virtual object is placed in an augmented reality userinterface, the virtual object is displayed at a fixed position relativeto a plane having the defined plane type. For example, a plane type thatcorresponds to virtual object 7110 (a virtual chair) is a horizontalphysical surface that can serve as a support surface for athree-dimensional representation of the virtual object in the augmentedreality view (e.g., a floor surface to support the chair). A plane typethat corresponds to virtual object 7118 (a photographic image) is avertical physical surface that can serve as a support surface for thethree-dimensional representation of the virtual object in the augmentedreality view (e.g., a vertical wall to hang the photographic image). Aplane type that corresponds to virtual object 7122 (a wall sconce) is avertical physical surface that can serve as a support surface for thethree-dimensional representation of the virtual object in the augmentedreality view (e.g., a vertical wall to hang the wall sconce). In FIG.7AK, in response to an input detected as described with regard to FIG.7AJ, virtual objects 7110, 7118 and 7122 are displayed in augmentedreality user interface 5022 in accordance with defined plane types forthe respective virtual objects. For example, virtual chair object 7110is displayed such that it appears to be supported by floor plane 5046detected in the physical environment, virtual photographic image object7118 is displayed such that it appears to be hanging on rear wall plane7132 detected in the physical environment, and virtual wall sconceobject 7122 is displayed such that it appears to be hanging on side wallplane 7134 detected in the physical environment.

In some embodiments, no plane type is defined for the selected virtualobjects and the virtual objects are displayed at a fixed positionrelative to a single plane (e.g., a default plane type, the firsthorizontal plane detected in the physical environment, and/or the firstplane of any type detected in the physical environment). In FIG. 7AL, inresponse to an input detected as described with regard to FIG. 7AJ,virtual objects 7110, 7118 and 7122 are displayed at fixed positionsrelative to floor plane 5046 such that the virtual objects appear to besupported by floor plane 5046.

FIGS. 7AM-7AQ illustrate manipulation of virtual objects to place in anaugmented reality user interface 5022.

In FIG. 7AM, an input (e.g., a rotation gesture) by contacts 7136 and7138 for changing the simulated orientation of virtual object 7118 isdetected. As contact 7136 moves along a path indicated by arrow 7140 andcontact 7138 moves along a path indicated by arrow 7142, virtual object7118 rotates about an axis that is perpendicular to vertical wall plane7132 relative to which virtual object 7118 is positioned, as illustratedin FIGS. 7AM-7AN. In FIGS. 7AN-7AO, contact 7136 continues to move alonga path indicated by arrow 7141, contact 7138 continues to move along apath indicated by arrow 7143, and virtual object 7110 continues torotate.

In FIG. 7AO, an input (e.g., a rotation gesture) by contacts 7144 and7146 for changing the simulated orientation of virtual object 7110 isdetected. As contact 7144 moves along a path indicated by arrow 7148 andcontact 7146 moves along a path indicated by arrow 7150, virtual object7110 rotates about an axis that is perpendicular to horizontal floorplane 5046 relative to which virtual object 7110 is positioned, asillustrated in FIGS. 7AO-7AP. In FIGS. 7AP-7AQ, contact 7144 continuesto move along a path indicated by arrow 7152, contact 7146 continues tomove along a path indicated by arrow 7154, and virtual object 7110continues to rotate.

FIGS. 8A-8K illustrate example user interfaces for displaying a promptto change a property of a media item that does not meet compatibilitycriteria for display in an augmented reality environment, in accordancewith some embodiments. The user interfaces in these figures are used toillustrate the processes described below, including the processes inFIGS. 9A-9C, 10A-10C, 11A-11D, 12A-12B, and 13A-13B. For convenience ofexplanation, some of the embodiments will be discussed with reference tooperations performed on a device with a touch-sensitive display system112. In such embodiments, the focus selector is, optionally: arespective finger or stylus contact, a representative pointcorresponding to a finger or stylus contact (e.g., a centroid of arespective contact or a point associated with a respective contact), ora centroid of two or more contacts detected on the touch-sensitivedisplay system 112. However, analogous operations are, optionally,performed on a device with a display 450 and a separate touch-sensitivesurface 451 in response to detecting the contacts on the touch-sensitivesurface 451 while displaying the user interfaces shown in the figures onthe display 450, along with a focus selector.

FIGS. 8A-8E illustrate handling of a media item that does not meetcompatibility criteria for display in an augmented reality environment.

In FIG. 8A, a file management user interface 8000 displays media itemsincluding AR-incompatible media item 8002. An input (e.g., a tap input)by contact 8004 is detected at a location that corresponds toAR-incompatible media item 8002. In response to the input,AR-incompatible media item 8002 is selected (e.g., as marked byselection indicator 8006).

In FIG. 8B, while AR-incompatible media item 8002 is selected, an input(e.g., a tap input) by contact 8007 is detected at a location thatcorresponds to share control 7130. In response to the input, sharinguser interface 6016 (described above with regard to FIG. 6C) isdisplayed with currently selected AR-incompatible media item 8002, asillustrated in FIG. 8C.

In FIG. 8C, an input (e.g., a tap input) by contact 8008 is detected ata location that corresponds to control 6024 for displaying one or moreselected objects, such as selected AR-incompatible media item 8002, in athree-dimensional viewing mode. In response to detecting the input, thedevice determines whether selected AR-incompatible media item 8002 has aproperty that does not meet compatibility criteria for display in anaugmented reality environment. Because selected AR-incompatible mediaitem 8002 has a property that does not meet compatibility criteria fordisplay in an augmented reality environment, the device displays aprompt panel 8012 including a prompt to change the property of the mediaitem (e.g., “The selected item is not AR compatible. Would you like toconvert the item to an AR-compatible format?”), as illustrated in FIG.8D. Prompt panel 8012 includes a control 8014 for converting selectedAR-incompatible media item 8002 to and a control 8016 for dismissingprompt panel 8012.

In FIG. 8D, an input (e.g., a tap input) by contact 8018 is detected ata location that corresponds to control 8014 for converting the selectedmedia item to an AR-compatible format. In response to the input,AR-incompatible media item 8002 is converted to an AR-compatible formatand the converted version of media item 8002 (e.g., virtual object 8003corresponding to the converted version of media item 8002) is displayedin augmented reality user interface 5022, as illustrated in FIG. 8F (orin object staging user interface 5004, as illustrated in FIG. 8E). Whilevirtual object 8003 corresponding to the converted version of media item8002 is displayed in object staging user interface 5004, an input (e.g.,a tap input) by contact 8020 is detected at a location that correspondsto the “AR” region of toggle control 5008. In response to the input,display of object staging user interface 5004 is replaced by display ofthe virtual object 8003 corresponding to the converted version of mediaitem 8002 in augmented reality user interface 5022, as illustrated inFIGS. 8E-8F.

FIGS. 8G-8K illustrate handling of a media item that meets compatibilitycriteria for display in an augmented reality environment.

In FIG. 8G, a file management user interface 8000 displays media itemsincluding AR-compatible media item 8022. An input (e.g., a tap input) bycontact 8028 is detected at a location that corresponds to AR-compatiblemedia item 8022. In response to the input, AR-compatible media item 8022is selected (e.g., as marked by selection indicator 8026).

In FIG. 8H, while AR-compatible media item 8022 is selected, an input(e.g., a tap input) by contact 8028 is detected at a location thatcorresponds to share control 7130. In response to the input, sharinguser interface 6016 (described above with regard to FIG. 6C) isdisplayed with currently selected AR-compatible media item 8022, asillustrated in FIG. 8I.

In FIG. 8I, an input (e.g., a tap input) by contact 8028 is detected ata location that corresponds to control 6024 for displaying one or moreselected objects, such as selected AR-compatible media item 8022, in athree-dimensional viewing mode. In response to detecting the input, thedevice determines whether selected AR-compatible media item 8022 has aproperty that does not meet compatibility criteria for display in anaugmented reality environment. Because selected AR-compatible media item8022 meets the compatibility criteria for display in an augmentedreality environment, the device displays a virtual object 8023 thatcorresponds to AR-compatible media item 8022 in augmented reality userinterface 5022, as illustrated in FIG. 8K (or in object staging userinterface 5004, as illustrated in FIG. 8J). While virtual object 8023that corresponds to AR-compatible media item 8022 is displayed in objectstaging user interface 5004, an input (e.g., a tap input) by contact8030 is detected at a location that corresponds to the “AR” region oftoggle control 5008. In response to the input, display of object staginguser interface 5004 is replaced by display of virtual object 8023 thatcorresponds to AR-compatible media item 8022 in augmented reality userinterface 5022, as illustrated in FIGS. 8J-8K.

FIGS. 9A-9C are flow diagrams illustrating method 900 of displaying avisual indication of one or more inputs that if performed would causecriteria to be satisfied for performing an action, in accordance withsome embodiments. Method 900 is performed at an electronic device (e.g.,device 300, FIG. 3, or portable multifunction device 100, FIG. 1A) witha display generation component (e.g., a display, a projector, a heads-updisplay, etc.) and one or more input devices (e.g., a touch-screendisplay that serves both as the display and the touch-sensitive surface,cameras, controllers, joysticks, buttons, etc.). In some embodiments,the display generation component is a touch-screen display and thetouch-sensitive surface is an input device that is on or integrated withthe display generation component. In some embodiments, the displaygeneration component is separate from one or more input devices. Someoperations in method 900 are, optionally, combined and/or the order ofsome operations is, optionally, changed.

This method relates to detecting an input while an environment thatincludes a virtual object is displayed. The virtual object is associatedwith an action (e.g., an animation). In response to the input, thedevice determines whether to perform the action or to display anindication of input that will cause the action to be performed.Displaying an indication of input that will cause the action associatedwith the virtual object to be performed provides improved visualfeedback to the user (e.g., by determining when the user is notproviding input that satisfies criteria for causing animation of thevirtual object and providing information about the required input to theuser). Providing improved visual feedback to the user increases theefficiency with which the user is able to cause the action associatedwith a virtual object to be performed, thereby enhancing the operabilityof the device, which, additionally, reduces power usage and improvesbattery life of the device by enabling the user to use the device morequickly and efficiently.

The device displays (902), via the display generation component (e.g.,touch-sensitive display system 112), at least a portion of anenvironment (e.g., a virtual reality environment, an augmented realityenvironment, or an object view environment that allows manipulation ofan object for placement in an augmented reality environment) thatincludes a virtual object that is associated with a first action (e.g.,an animation sequence that animates at least a portion of the virtualobject) that is triggered based on satisfaction of a first set ofcriteria. For example, FIG. 5F illustrates a virtual object 5002displayed in augmented reality user interface 5022. As shown in FIGS.5E-5F, augmented reality user interface 5022 displays an augmentedreality environment that includes virtual object 5002 and a physicalenvironment as captured by one or more cameras (e.g., one or morecameras of camera module 143) of device 100. Virtual object 5002 isassociated with an animation sequence, as illustrated in FIGS. 5H-5I.

While displaying, by the display generation component, the portion ofthe environment, the device detects (904) a first input. For example, inFIG. 5G, a tap input is detected at a location that corresponds tovirtual object 5002. In FIGS. 5J-5K, a swipe input is detected at alocation that corresponds to virtual object 5002.

In response to detecting the first input (906): in accordance with adetermination that the first input satisfies the first set of criteria(e.g., the first set of criteria are satisfied by a tap input, a pressinput, detection of a gaze of a user of the electronic device directedto a location that corresponds to the virtual object, an input thatcauses the virtual object to be displayed at a designated portion of afield of view of one or more cameras, and/or movement of at least aportion of the electronic device to within a defined distance of aposition of the displayed of the virtual object), the device performsthe first action (e.g., animating at least a portion of the virtualobject in accordance with the animation sequence associated with thevirtual object) and, in accordance with a determination that the firstinput does not satisfy the first set of criteria but instead satisfies asecond set of criteria (e.g., the second set of criteria are satisfiedby a swipe input for moving the virtual object, a swipe input forrotating the virtual object, a pinch input for reducing the displayedsize of the virtual object, a de-pinch input for increasing thedisplayed size of the virtual object, detection of a gaze of a user ofthe electronic device directed to a location that does not correspond tothe virtual object, an input that causes the virtual object to bedisplayed at a location that is not within the designated portion of afield of view of one or more cameras, and/or determination that theelectronic device is not within a defined distance of a position of thedisplayed of the virtual object), the device forgoes performing thefirst action and instead display a first visual indication of one ormore inputs (e.g., a text prompt indicating the one or more inputsand/or an animation illustrating the one or more inputs) that ifperformed would cause the first set of criteria to be satisfied.

For example, in response to an input that satisfies a first set ofcriteria (e.g., the tap input described with regard to FIG. 5G), a firstaction is performed (e.g., playback of an animation sequence occurs, asdescribed with regard to FIGS. 5H-5I). In response to an input that doesnot satisfy the first set of criteria but instead satisfies a second setof criteria (e.g., the swipe input described with regard to FIGS.5J-5K), the device forgoes performing the first action and insteaddisplays a first visual indication of one or more inputs that ifperformed would cause the first set of criteria to be satisfied (e.g.,the device displays visual indication 5028 indicating, “Tap on the boxto view animation!” as described with regard to FIG. 5K). In someembodiments, an audio indication of one or more inputs that if performedwould cause the first set of criteria to be satisfied is provided inaddition to or in lieu of the visual indication.

In some embodiments, the first set of criteria include (908) criteriathat are satisfied in accordance with a determination that at least aportion of the electronic device (e.g., a respective input device of theone or more input devices, the display generation component, and/or oneor more cameras) moves to a position that is within a defined distanceof a displayed position of the virtual object. In some embodiments, thefirst visual indication of one or more inputs that if performed wouldcause the first set of criteria to be satisfied includes a direction tomove the electronic device to a position that is within the defineddistance of the displayed position of the virtual object (e.g., “Movethe device closer to the box.”). For example, as described with regardto FIGS. 5AD-5AH, a first set of criteria for performing an action aresatisfied in accordance with a determination that device 100 has movedto a position that is within a defined distance of a displayed positionof virtual object 5002. In FIGS. 5AD-5AE, the device is not within thedefined distance of the displayed position of virtual object 5002, and avisual indication 5060 of one or more inputs that if performed wouldcause the first set of criteria to be satisfied is displayed. In FIG.5AF, the device has moved to within the defined distance of thedisplayed position of virtual object 5002 and an animation sequence isperformed, as described with regard to FIGS. 5AF-5AH. Determiningwhether to perform an action associated with the virtual object ordisplay an indication of input that will cause the action to beperformed, depending on whether the device moves to a position that iswithin a defined distance of a displayed position of the virtual object,provides improved visual feedback to the user (e.g., by determining whenthe device is not sufficiently near to the virtual object and providinginformation indicating that the user must move closer to the displayedposition of the virtual object to activate the animation of the virtualobject). Providing improved visual feedback to the user increases theefficiency with which the user is able to cause the action associatedwith a virtual object to be performed, thereby enhancing the operabilityof the device, which, additionally, reduces power usage and improvesbattery life of the device by enabling the user to use the device morequickly and efficiently.

In some embodiments, the one or more input devices include (910) atouch-screen display (e.g., touch-sensitive display system 112) and thefirst set of criteria include criteria that are satisfied in accordancewith a determination that the first input is a tap input by a contact onthe touch-screen display at a location that corresponds to the virtualobject (e.g., as described with regard to FIGS. 5G-5I). In someembodiments, the first visual indication includes a direction to tap alocation that corresponds to the virtual object (e.g., “Tap on thebox.”). Determining whether to perform an action associated with thevirtual object or display an indication of input that will cause theaction to be performed, depending on whether the input is a tap input bya contact on a touch-screen display at a location that corresponds tothe virtual object, provides improved visual feedback to the user (e.g.,by determining whether the input is a tap input and providinginformation indicating that the user must tap the virtual object toactivate the animation of the virtual object). Providing improved visualfeedback increases the efficiency with the user is able to cause theaction associated with a virtual object to be performed, therebyenhancing the operability of the device, which, additionally, reducespower usage and improves battery life of the device by enabling the userto use the device more quickly and efficiently.

In some embodiments, the one or more input devices include (912) asensor (e.g., one or more cameras of camera module 143) for detecting adirection of a gaze of a user of the electronic device and the first setof criteria include criteria that are satisfied in accordance with adetermination that the gaze of the user is directed toward a locationthat corresponds to the virtual object (e.g., a location within adesignated portion of the virtual object, a location at any position onthe virtual object, and/or a location at a position within a defineddistance of the virtual object). For example, as described with regardto FIGS. 5L-1 and 5L-2, 5M-1 and 5M-2, 5N-1 and 5N-2, and 5O-1 and 5O-2,a first set of criteria for performing an action are satisfied inaccordance with a determination that a gaze of a user is directed towarda location that corresponds to virtual object 5002. In FIGS. 5L-1 and5L-2, gaze 5034 of user 5030 is not directed toward a displayed locationof virtual object 5002, and a visual indication 5036 of one or moreinputs that if performed would cause the first set of criteria to besatisfied is displayed. In FIGS. 5M-1 and 5M-2, gaze 5038 of user 5030is directed toward a displayed location of virtual object 5002 and ananimation sequence is performed, as described with regard to FIGS. 5M-1and 5M-2, FIGS. 5N-1 and 5N-2 and 5O-1 and 5O-2. In some embodiments,the first visual indication includes a direction to direct the user'sgaze toward a location that corresponds to the virtual object (e.g.,“Look at the box.”). Determining whether to perform an action associatedwith the virtual object or display an indication of input that willcause the action to be performed, depending on whether the gaze of theuser is directed toward a location that corresponds to the virtualobject, enables the performance of multiple different types ofoperations in response to an input. Enabling the performance of multipledifferent types operations in response to an input increases theefficiency with which the user is able to perform these operations,thereby enhancing the operability of the device, which, additionally,reduces power usage and improves battery life of the device by enablingthe user to use the device more quickly and efficiently.

In some embodiments, the electronic device includes one or more cameras(e.g., one or more rear facing cameras on a side of the device oppositefrom the display generation component), the displayed environmentincludes at least a portion of the field of view of the one or morecameras, the device detects (914) movement of the electronic device(e.g., lateral movement and/or rotation of the electronic device) thatadjusts the field of view of the one or more cameras and, in response todetecting the movement of the electronic device, the device adjusts thevirtual object in accordance with a fixed spatial relationship (e.g.,orientation and/or position) between the virtual object and a respectiveplane in the field of view of the one or more cameras (e.g., the virtualobject is displayed with an orientation and a position on the displaysuch that a fixed angle between the virtual object and the plane ismaintained (e.g., the virtual object appears to stay at a fixed locationon the plane) as the field of view of the one or more cameras isadjusted in response to detected movement of the electronic device). Forexample, as described with regard to FIG. 5E, augmented reality userinterface 5022 displays a physical environment 5018 as captured in thefield of view of one or more cameras. As described with regard to FIGS.5S-5U, as the device moves such that the field of view of the one ormore cameras is adjusted, virtual object 5002 moves in accordance with afixed spatial relationship between virtual object 5002 and a plane thatcorresponds to upper surface of physical table 5020. Adjusting thevirtual object in accordance with a fixed spatial relationship betweenthe virtual object and a plane in the field of view of the one or morecameras in response to movement of the electronic device providesimproved visual feedback to the user (e.g., by providing an indicationthat the virtual object has been placed at a fixed position relative tothe plane). Providing improved visual feedback to the user enhances theoperability of the device and makes the user-device interface moreefficient, which, additionally, reduces power usage and improves batterylife of the device by enabling the user to use the device more quicklyand efficiently.

In some embodiments, the first set of criteria include (916) criteriathat are satisfied in accordance with a determination that the virtualobject is displayed within a designated portion of the field of view ofthe one or more cameras (e.g., the virtual object is fully visible inthe displayed portion of the field of view of the one or more cameras,the virtual object is substantially centered in the displayed portion ofthe field of view of the one or more cameras, the virtual object isdisplayed within a designated portion of the field of view of the one ormore cameras, and/or at least a predetermined portion of the virtualobject displayed in the field of view of the cameras). For example, InFIGS. 5S-5T, virtual object 5002 is not fully displayed within adesignated portion (e.g., circle 5052) of the field of view of the oneor more cameras displayed in augmented reality user interface 5022, anda visual indication 5054 of one or more inputs that if performed wouldcause the first set of criteria to be satisfied is displayed. In FIG.5U, virtual object 5002 is fully displayed within circle 5052 and ananimation sequence is performed, as described with regard to FIGS.5U-5W. In some embodiments, the first visual indication includes adirection to move the device such that the virtual object is displayedwithin the designated portion of the field of view of the one or morecameras (e.g., “Move the device to view the box.”). In some embodiments,the first set of criteria require that multiple criteria (e.g., gaze atvirtual object, tap input on virtual object, proximity to virtualobject, and/or virtual object in field of view) are satisfied (e.g., asdescribed with regard to FIGS. 5X-5AC). In some embodiments, inaccordance with a determination that virtual object display criteria aremet (e.g., the virtual object is fully visible in the displayed portionof the field of view of the one or more cameras, the virtual object issubstantially centered in the displayed portion of the field of view ofthe one or more cameras, the virtual object is displayed within adesignated portion of the field of view of the one or more cameras,and/or at least a predetermined portion of the virtual object displayedin the field of view of the cameras), the first visual indication is notdisplayed. In some embodiments, the virtual object has a boundary (e.g.,an outline of virtual object 5002 or an oval or box that circumscribesvirtual object 5002), performing the action (e.g., animation of thevirtual object) causes at least a portion of the virtual object toextend beyond the first boundary, a second boundary corresponds to theextended range of the first respective virtual object during theanimation, and the first set of criteria include criteria that aresatisfied in accordance with a determination that the second boundary isfully displayed in the displayed portion of the field of view of thecameras of the device. Determining whether to perform an actionassociated with the virtual object or display an indication of inputthat will cause the action to be performed, depending on whether thevirtual object is displayed within a designated portion of the field ofview of the one or more cameras, enables the performance of multipledifferent types of operations in response to an input. Enabling theperformance of multiple different types operations in response to aninput increases the efficiency with which the user is able to performthese operations, thereby enhancing the operability of the device,which, additionally, reduces power usage and improves battery life ofthe device by enabling the user to use the device more quickly andefficiently.

In some embodiments, the second set of criteria include (918) criteriathat are met in accordance with a determination that the first input isan input for moving the virtual object (e.g., a swipe input thatincludes movement of a contact along a path on a touch-screen display).For example, an input for moving virtual object 5002 is described withregard to FIGS. 5J-5K. Displaying an indication of input that will causethe action to be performed in response to an input for moving thevirtual object provides improved visual feedback to the user (e.g., byproviding information to the user about a different input that isrequired for animating the virtual object). Providing visual feedback tothe user enhances the operability of the device (e.g., by helping theuser to provide proper inputs and reducing unnecessary additional inputsfor interacting with the virtual object), which, additionally, reducespower usage and improves battery life of the device by enabling the userto use the device more quickly and efficiently.

In some embodiments, the device receives (920) a request to display thevirtual object at a first location in the environment, and, in responseto the request to display the virtual object at the first location inthe environment: in accordance with a determination that a surface sizethat corresponds to the first location does not meet size compatibilitycriteria for the virtual object, the device displays a visual indicationthat provides information indicating that the first location does notmeet the size compatibility criteria (and forgoes displaying the virtualobject at the first location in the environment) and in accordance witha determination that the surface size that corresponds to the firstlocation meets the size compatibility criteria for the virtual object(e.g., the surface size that corresponds to the first location issmaller (e.g., by a predefined amount) than a size of the virtualobject), the device displays the virtual object at the first location inthe environment. For example, as described with regard to FIGS. 5Q-5R,in response to a request to display virtual object 5002 at an uppersurface of physical lamp 5004 (e.g., input that moves virtual object5002 along a path indicated by arrow 5048 in FIG. 5Q), a visualindication 5050 provides information indicating that the first locationdoes not meet the size compatibility criteria (e.g., informationincluding text that indicates “The object cannot be placed on thissurface!”) because a size of the upper surface of physical lamp 5044does not meet size compatibility criteria for virtual object 5002. Insome embodiments, the virtual object has a size property (e.g., thatcorresponds to a real world size of the object) that indicates a size(e.g., length, width, radius, and/or area) of at least a portion of thevirtual object. For example, the size property indicates a size of aportion of the virtual object (e.g., a size of a surface of the virtualobject that will be placed at a fixed position relative to (e.g.,facing) a surface in the physical environment). The first location is,for example, an upper surface of a table, a wall, a floor, or a ceiling.Displaying a visual indication in response to determining that alocation where the user requested to display the virtual object does notmeet size compatibility criteria provides the user with improved visualfeedback (e.g., by providing an indication of why the virtual object isnot being placed at the location and/or how to place the virtual objectat a location that meets size compatibility criteria). Providingimproved visual feedback enhances the operability of the device (e.g.,by helping the user to provide proper inputs and reducing unnecessaryadditional inputs for placing the virtual object) and makes theuser-device more efficient, which, additionally, reduces power usage andimproves battery life of the device by enabling the user to use thedevice more quickly and efficiently.

In some embodiments, the movement of the electronic device is detected(922) while (and/or after) performing the first action, adjusting thevirtual object in response to detecting the movement of the electronicdevice in accordance with the fixed spatial relationship between thevirtual object and the respective plane causes the virtual object tomove at least partially beyond the displayed portion of the field ofview of the one or more cameras, and, while the virtual object is atleast partially beyond the displayed portion of the field of view of theone or more cameras, the device displays a second visual indication(e.g., the same as or different from the first visual indication) of oneor more inputs that if performed would cause the first set of criteriato be satisfied. Displaying an indication of inputs that would cause thefirst set of criteria to be satisfied while the virtual object is beyondthe displayed field of view of the one or more cameras provides the userwith improved visual feedback (e.g., to indicate that the user must movethe device such that the virtual object is visible (e.g., fully visible)within the field of view of the one or more cameras). Providing improvedvisual feedback enhances the operability of the device and makes theuser-device interface more efficient (e.g., by helping the user toprovide proper inputs, reducing unnecessary additional inputs forcausing the first set of criteria to be satisfied), which, additionally,reduces power usage and improves battery life of the device by enablingthe user to use the device more quickly and efficiently.

In some embodiments, the electronic device includes one or more camerasand, in accordance with a determination that the first input correspondsto a request to display the virtual object in an augmented realityenvironment (e.g., an input at a location that corresponds to the “AR”region of toggle control 5008, as described with regard to FIG. 5D) thedevice replaces (924) display of the environment that includes thevirtual object with display of at least a portion of the field of viewof the one or more cameras (e.g., as described with regard to FIGS.5E-5F). In some embodiments, the environment displayed by the displaygeneration component includes a background other than an image of aphysical environment surrounding the device (e.g., the background of thefirst user interface region is a preselected background color/pattern,or a background image that is distinct from an output image concurrentlycaptured by the one or more cameras and distinct from live content in afield of view of the one or more cameras) and, in response to a requestto display the virtual object in an augmented reality environment, thedisplay of the environment is replaced with display of the physicalenvironment surrounding the device (e.g., as captured in the field ofview of the one or more cameras). Either performing an action associatedwith a virtual object, displaying an indication of input that will causethe action to be performed, or replacing display of an environment thatincludes a virtual object with a display of the virtual object with afield of view of one or more cameras in response to an input enables theperformance of multiple different types of operations in response to theinput. Enabling the performance of multiple different types ofoperations in response to an input increases the efficiency with whichthe user is able to perform these operations, thereby enhancing theoperability of the device, which, additionally, reduces power usage andimproves battery life of the device by enabling the user to use thedevice more quickly and efficiently.

In some embodiments, the second set of criteria include (926) criteriathat are met in accordance with a determination that the first input isan input for altering a characteristic (e.g., an orientation and/orsize) the virtual object. In some embodiments, an input for altering acharacteristic includes a swipe input (e.g., movement of a contact alonga path on the touch-screen display). For example, the input for alteringa characteristic of the virtual object is an input detected whilevirtual object 5002 is displayed in object staging user interface 5004for rotating virtual object 5002, as described with regard to FIGS.5B-5C. In some embodiments, the swipe input changes the orientation ofthe virtual object (e.g., by rotating the virtual object about one ormore axes). In some embodiments, an input for altering a characteristicincludes a pinch and/or de-pinch input (e.g., movement of two contactsalong respective paths on the touch-screen display that bring thecontacts closer together (pinch) or further apart (de-pinch)). In someembodiments, the pinch and/or de-pinch input changes the displayed sizeof the virtual object. In some embodiments, the first input thatsatisfies the first set of criteria is an input (e.g., a tap input at alocation that corresponds to virtual object 5002) detected while virtualobject 5002 is displayed in object staging user interface 5004 (e.g.,causing an animation sequence associated with virtual object 5002 tooccur while virtual object 5002 is displayed in object staging userinterface 5004). Either performing an action associated with a virtualobject, displaying an indication of input that will cause the action tobe performed, or replacing display of an environment that includes avirtual object with a display of the virtual object with a field of viewof one or more cameras in response to an input enables the performanceof multiple different types of operations in response to the input.Enabling the performance of multiple different types of operations inresponse to an input increases the efficiency with which the user isable to perform these operations, which, additionally, reduces powerusage and improves battery life of the device by enabling the user touse the device more quickly and efficiently.

In some embodiments, the first visual indication of the one or moreinputs that if performed would cause the first set of criteria to besatisfied includes (928) a text prompt (e.g., a text prompt indicating“Tap on the box to view animation!” as described with regard to visualindication 5028 in FIG. 5K). Displaying a text prompt indicating inputthat will cause the action associated with the virtual object to beperformed provides improved visual feedback to the user (e.g., byproviding a text that indicates the required input to the user).Providing improved visual feedback to the user increases the efficiencywith which the user is able to cause the action associated with avirtual object to be performed, thereby enhancing the operability of thedevice, which, additionally, reduces power usage and improves batterylife of the device by enabling the user to use the device more quicklyand efficiently

In some embodiments, the device detects (930) a second input thatsatisfies the first set of criteria while displaying the first visualindication of the one or more inputs that if performed would cause thefirst set of criteria to be satisfied and, in response to detecting thesecond input that satisfies the first set of criteria, the device ceasesto display the first visual indication. For example, as described withregard to FIGS. 5AD-5AF, an input that satisfies a first set of criteriafor performing an action is detected while visual indication 5060 isdisplayed, and, in accordance with a determination that device 100 hasmoved to a position that is within a defined distance of a displayedposition of virtual object 5002 (as described with regard to FIG. 5AF),the device ceases to display visual indication 5060. Ceasing to displaythe visual indication of input that will cause the action to beperformed, in response to detecting input that satisfies the first setof criteria (e.g., without requiring further user input to dismiss thefirst visual indication) reduces the number of inputs needed to performan operation. Reducing the number of inputs needed to perform anoperation enhances the operability of the device and makes theuser-device interface more efficient, which, additionally, reduces powerusage and improves battery life of the device by enabling the user touse the device more quickly and efficiently.

In some embodiments, performing the first action includes (932)animating at least a portion of the virtual object (e.g., animating thevirtual object 5002 such that the lid of the box opens, as describedwith regard to FIGS. 5G-5I). Animating at least a portion of the virtualobject provides the user with improved visual feedback (e.g., toindicate that the provided input satisfies the criteria for animatingthe object). Providing improved visual feedback enhances the operabilityof the device and makes the user-device interface more efficient (e.g.,by helping the user to provide proper inputs and reducing user mistakeswhen operating/interacting with the device), which, additionally,reduces power usage and improves battery life of the device by enablingthe user to use the device more quickly and efficiently.

In some embodiments, the device displays (934) (e.g., during and/orafter performing the first action) a video play head control foradjusting playback of an animated sequence that animates the portion ofthe virtual object. For example, as described with regard to FIGS.5AI-5AK, input that moves video play head control 5066 adjusts playbackof an animated sequence that animates the lid of virtual object 5002.Displaying a video play head control for adjusting playback of ananimated sequence that animates the portion of the virtual objectprovides the user with improved visual feedback (e.g., to indicate thatplayback of the virtual object is adjustable). Providing improved visualfeedback enhances the operability of the device and makes theuser-device interface more efficient, which, additionally, reduces powerusage and improves battery life of the device by enabling the user touse the device more quickly and efficiently.

In some embodiments, the electronic device includes (936) an audiogeneration component, performing the first action includes generating,by the audio generation component, a first audio channel and a secondaudio channel, and the device adjusts an amplitude of the first audiochannel and an amplitude of the second audio channel in accordance witha displayed position of the virtual object in the environment. In someembodiments, spatial audio is used to provide the user of an indicationof a distance and/or position of the virtual object in the environmentrelative to a (real or virtual) position of the user in the environment.For example, as the distance between the virtual object and the userdecreases, the amplitude of the first audio channel and the amplitude ofthe second audio channel increase, and as the virtual object moves fromleft of the user to right of the user, an amplitude of the left audiochannel is decreased and an amplitude of the right audio channel isincreased. Adjusting an amplitude of a first audio channel and anamplitude of a second audio channel based on a displayed position of thevirtual object in the environment provides the user with aural feedback(e.g., to indicate the distance and the location of the virtual objectin the environment relative to the position of the user in theenvironment). Providing aural feedback enhances the operability of thedevice and makes the user-device interface more efficient (e.g., byhelping the user to locate a virtual object that is not currently withinthe displayed field of view of the one or more cameras), which,additionally, reduces power usage and improves battery life of thedevice by enabling the user to use the device more quickly andefficiently.

It should be understood that the particular order in which theoperations in FIGS. 9A-9C have been described is merely an example andis not intended to indicate that the described order is the only orderin which the operations could be performed. One of ordinary skill in theart would recognize various ways to reorder the operations describedherein. Additionally, it should be noted that details of other processesdescribed herein with respect to other methods described herein (e.g.,methods 1000, 1100, 1200, and 1400) are also applicable in an analogousmanner to method 900 described above with respect to FIGS. 9A-9C. Forexample, the contacts, inputs, gestures, virtual objects, environments,user interface regions, user interface objects, fields of view,movements, and/or animations described above with reference to method900 optionally have one or more of the characteristics of the contacts,inputs, gestures, media items, virtual objects, environments, userinterface regions, user interface objects, fields of view, movements,and/or animations described herein with reference to other methodsdescribed herein (e.g., methods 1000, 1100, 1200, and 1400). Forbrevity, these details are not repeated here.

FIGS. 10A-10C are flow diagrams illustrating method 1000 of displaying amedia item in a user interface that includes content of at least aportion of a field of view of one or more cameras, in accordance withsome embodiments. Method 1000 is performed at an electronic device(e.g., device 300, FIG. 3, or portable multifunction device 100, FIG.1A) with a display generation component (e.g., a display, a projector, aheads-up display, etc.), one or more input devices (e.g., a touch-screendisplay that serves both as the display and the touch-sensitive surface,cameras, controllers, joysticks, buttons, etc.), and one or cameras(e.g., one or more rear-facing cameras on a side of the device oppositefrom the display and the touch-sensitive surface). In some embodiments,the display generation component is a touch-screen display and thetouch-sensitive surface is an input device that is on or integrated withthe display generation component. In some embodiments, the displaygeneration component is separate from one or more input devices. Someoperations in method 1000 are, optionally, combined and/or the order ofsome operations is, optionally, changed.

This method relates to displaying a representation of a media item in auser interface that includes content of at least a portion of a field ofview of one or more cameras. Displaying the media item in a userinterface that includes content of at least a portion of a field of viewof one or more cameras reduces the number of inputs needed to perform anoperation (e.g., by providing a control in an existing sharing userinterface for displaying the representation of the media item in aninterface that includes content of at least a portion of a field of viewof one or more cameras). Reducing the number of inputs needed to performan operation enhances the operability of the device and makes theuser-device interface more efficient (e.g., by allowing the user todisplay a representation of the media item in interface that includescontent of at least a portion of a field of view of one or more cameraswithout needing to provide inputs and/or access external applications toconvert the media item for display in an interface that includes contentof at least a portion of a field of view of one or more cameras), which,additionally, reduces power usage and improves battery life of thedevice by enabling the user to use the device more quickly andefficiently.

The device displays (1002), by the display generation component (e.g.,touch-sensitive display system 112): a first user interface region(e.g., photo viewing interface 6004 of a photo management application,as described with regard to FIG. 6A), a first representation of a firstmedia item (e.g., a photograph (such as photo 6002), an image, a stillimage that corresponds to a video, album art that corresponds to musiccontent, or a thumbnail representation thereof) in the first userinterface region, and a first user interface object (e.g., control 6008)for displaying a sharing user interface (e.g., an interface thatprovides a plurality of affordances corresponding to various sharingmethods for the first media item and/or other media items).

While displaying the first user interface region, the device detects(1004) a first input corresponding to the sharing user interface object.For example, the device detects an input at a location that correspondsto control 600) for displaying a sharing user interface as describedwith regard to FIG. 6B.

In response to detecting the first input corresponding to the first userinterface object, the device displays (1006) the sharing user interface(e.g., overlaid over the first user interface or replacing display ofthe first user interface) that includes a second user interface objectfor initiating a process for displaying a second representation of thefirst media item in an augmented reality environment (e.g., control 6024for displaying one or more selected objects in an objectthree-dimensional viewing mode). For example, in response to the inputdescribed with regard to FIG. 6B, the device displays sharing userinterface 6016 that includes control 6024, as described with regard toFIG. 6C.

While the sharing user interface is displayed, the device detects (1008)a sequence of one or more inputs including selection of the second userinterface object. For example, as described with regard to FIG. 6C,while sharing user interface 6016 is displayed, an input is detected ata location that corresponds to control 6024.

In response to detecting the sequence of one or more inputs includingselection of the second user interface object, the device (ceases todisplay the sharing user interface and/or the first user interfaceregion, and) displays (1010) the second representation of the firstmedia item in a second user interface (e.g., augmented reality userinterface 5022) that includes content of at least a portion of the fieldof view of the one or more cameras (e.g., one or more cameras of cameramodule 143). For example, in response to the input described with regardto FIG. 6C, the device displays photo 6002 in augmented reality userinterface 5022 that includes at least a portion of the field of view ofone or more cameras of device 100, as described with regard to FIGS.6D-6E. In some embodiments, the second representation of the first mediaitem is displayed with a simulated depth (e.g., to give a flat image anappearance of depth in the environment). For example, as described withregard to FIG. 6E, image 6002 is displayed with a simulated backing 6034that adds an appearance of depth in the environment to image 6002. Insome embodiments, the second representation of the first media item ismodified to include a simulated reflectivity (e.g., so that the imageappears to be glossy) and/or increased luminance (e.g., so that theimage appears to be shiny).

In some embodiments, the displayed second representation of the firstmedia item overlays (1012) a respective plane (e.g., a wall, a floor, ora ceiling) in the field of view of the one or more cameras. For example,in FIG. 6E, photo 6002 overlays wall 6032 of physical environment 5018as detected in the field of view of the one or more cameras of device100. Displaying a media item representation overlaying a plane in anenvironment provides improved visual feedback to the user (e.g.,allowing the user to see the distance and location of the media item inan environment). Providing improved visual feedback enhances theoperability of the device and makes the user-device interface moreefficient (e.g., providing easy access and manipulation of the mediaitem to the user), which, additionally, reduces power usage and improvesbattery life of the device by enabling the user to use the device morequickly and efficiently.

In some embodiments, while displaying the second representation of thefirst media item in the second user interface, the device detects (1014)an input for rotating the second representation of the first media item(e.g., an input by two contacts that move (e.g., clockwise orcounterclockwise) along a rotational path). In response to the input,the device rotates the second representation of the first media itemabout an axis that is normal to the respective plane in the field ofview of the one or more cameras over which the first media item isdisplayed. For example, as described with regard to FIGS. 6W-6Y, inresponse to an input for rotating photo 6046 about an axis that isnormal to wall 6032 over which photo 6046 is displayed, the devicerotates photo 6046 about the axis that is normal to wall 6032. In someembodiments, in accordance with a determination that the input forrotating the second representation of the first media item is a requestto rotate the second representation of the first media item about anunavailable axis (e.g., an axis that is not normal to the respectiveplane in the field of view of the one or more cameras over which thefirst media item is displayed), a visual indication is displayed (e.g.,to indicate that rotation is not permitted about the unavailable axis.)For example, as described with regard to FIG. 6Z, in response to aninput for rotating photo 6046 about axis 6106 that is parallel to wall6032 over which photo 6046 is displayed, the device displays visualindication 6098 indicating that rotation is not permitted about axis6106. Rotating the media item representation about an axis that isnormal to the plane in the field of view of the one or more cameras inresponse to an input reduces the number of inputs needed to perform anoperation (e.g., by allowing the user to directly interact with thedisplayed representation of the media item as displayed in the augmentedreality user interface, rather than requiring the user to indicate apreferred rotation in a different context before the media item isdisplayed in the augmented reality user interface). Reducing the numberof inputs needed to perform an operation enhances the operability of thedevice, and makes the user-device interface more efficient, which,additionally, reduces power usage and improves battery life of thedevice by enabling the user to use the device more quickly andefficiently.

In some embodiments, the device detects (1016) first movement of theelectronic device (e.g., lateral movement and/or rotation of theelectronic device) that adjusts the field of view of the one or morecameras (e.g., as illustrated in FIGS. 6E-6F) and, in response todetecting the movement of the electronic device, the device adjusts thesecond representation of the first media item (e.g., photo 6002) inaccordance with a fixed spatial relationship (e.g., orientation and/orposition) between the second representation of the first media item andthe respective plane (e.g., wall 6032) in the field of view of the oneor more cameras. Adjusting the media item representation in accordancewith a fixed spatial relationship with respect to the field of view ofthe cameras provides improved visual feedback to the user (e.g., toindicate the fixed spatial relationship of the media item representationrelative to the detected plane in the physical environment). Providingimproved visual feedback enhances the operability of the device andmakes the user-device interface more efficient (e.g., by helping theuser to provide proper inputs and reducing use mistakes when interactingwith the device), which, additionally, reduces power usage and improvesbattery life of the device by enabling the user to use the device morequickly and efficiently.

In some embodiments, the device detects (1018) selection of a firstrepresentation of a second media item (e.g., while the first userinterface region is displayed or while the sharing user interface isdisplayed) and, in response to detecting the sequence of one or moreinputs including selection of the second user interface object, thedevice displays a second representation of the second media item in thesecond user interface that includes the content of at least a portion ofthe field of view of one or more cameras (e.g., the secondrepresentation of the first media item and the second representation ofthe second media item are concurrently displayed in a portion of thefield of view of one or more cameras). For example, as described withregard to FIGS. 6I-6J, inputs for selecting a first media item (photo6002) and a second media item (photo 6046) is detected. In response todetecting a sequence of inputs that includes the inputs for selectingphotos 6002 and 6046 (and inputs for displaying photos 6002 and 6046 inaugmented reality user interface 5022, as described with regard to FIGS.6L-6M), photos 6002 and 6046 are displayed in augmented reality userinterface 5022 that includes content of at least a portion of the fieldof view of one or more cameras, as described with regard to FIGS. 6N-6O.In some embodiments, the second representation of the first media itemand the second representation of the second media item are displayedwith fixed spatial relationships to plane (e.g., overlaying the sameplane) in the field of view of the one or more cameras. Displayingmultiple media items in a user interface that includes the field of viewof one or more cameras in response to detecting the sequence of one ormore inputs reduces the number of inputs needed to perform an operation(e.g., by allowing the user to select multiple media items using asingle selection user interface and simultaneously place the multipleselected items). Reducing the number of inputs needed to perform anoperation enhances the operability of the device, which additionally,reduces power usage and improves battery life of the device by enablingthe user to use the device more quickly and efficiently.

In some embodiments, the selection of the second media item is detected(1020) while the sharing user interface is displayed. For example, thefirst media item is selected by the first input corresponding to thesharing user interface object (e.g., when the first input is receivedwhile the first media item is displayed in the first user interfaceregion) and the second media item is selected by an additional inputreceived (e.g., among the sequence of one or more inputs includingselection of the second user interface object) at a locationcorresponding to the second media item while the sharing user interfaceis displayed. Displaying multiple media items in a user interface thatincludes the field of view of one or more cameras in response todetecting the sequence of one or more inputs reduces the number ofinputs needed to perform an operation (e.g., by allowing the user toprovide a single input at a sharing user interface object tosimultaneously place multiple selected items). Reducing the number ofinputs needed to perform an operation enhances the operability of thedevice, which additionally, reduces power usage and improves batterylife of the device by enabling the user to use the device more quicklyand efficiently.

In some embodiments, while displaying the first user interface region,the device displays (1022) the first representation of the second mediaitem and the selection of the second media item is detected while thefirst user interface region is displayed. For example, the first mediaitem (e.g., photo 6002) is selected by a first input (e.g., a tap input)detected at a location corresponding to the first media item (e.g., asdescribed with regard to FIG. 6I) while the first user interface region(e.g., a photo selection interface 6036) is displayed and the secondmedia item (e.g., photo 6046) is selected by a second input (e.g., a tapinput) detected at a location corresponding to the second media item(e.g., as described with regard to FIG. 6J) while the first userinterface region is displayed (e.g., prior to detection of the firstinput corresponding to the sharing user interface object). Displayingmultiple media items in a user interface that includes the field of viewof one or more cameras in response to detecting the sequence of one ormore inputs reduces the number of inputs needed to perform an operation(e.g., by allowing the user to select multiple media items using asingle selection user interface and simultaneously place the multipleselected items). Reducing the number of inputs needed to perform anoperation enhances the operability of the device, which additionally,reduces power usage and improves battery life of the device by enablingthe user to use the device more quickly and efficiently.

In some embodiments, selection of the first media item is detected(1024) at a first time (e.g., by the first input corresponding to thesharing user interface object (e.g., when the first input is receivedwhile the first media item is displayed in the first user interfaceregion) or by a first selection input (e.g., a tap input at a locationthat corresponds to the first media item) detected prior to the firstthe first input corresponding to the sharing user interface object). Forexample, selection of the first representation of photo 6002 is detected(e.g., as described with regard to FIG. 6I) at a first time. Selectionof the second media item is detected at a second time (e.g., by aselection input detected while the sharing user interface is displayedor by a second selection input (e.g., a tap input at a location thatcorresponds to the second media item) detected prior to detection of thefirst input corresponding to the sharing user interface object). Forexample, selection of the first representation of photo 6046 is detected(e.g., as described with regard to FIG. 6J) at a second time. Displayingthe second representation of the first media item in the second userinterface and displaying the second representation of the second mediaitem in the second user interface includes: in accordance with adetermination that the first time is prior to the second time,displaying the second representation of the first media item and thesecond representation of the second media item with a first order (e.g.,the first media item is displayed to the left of (or above, or in frontof) the second media item (e.g. overlaying a plane in the field of viewof the one or more cameras)) and in accordance with a determination thatthe second time is prior to the first time, displaying the secondrepresentation of the first media item and the second representation ofthe second media item with a second order (e.g., the first media item isdisplayed to the right of (or below, or behind) the second media item)that is distinct from the first order. For example, because the firsttime at which the first representation of photo 6002 was selected isprior to the second time at which the first representation of photo 6046was selected, the second representation of photo 6002 is displayed tothe left of the second representation of photo 6046 in FIG. 6O. If thefirst representation of photo 6046 was selected prior to the firstrepresentation of photo 6046, the second representation of photo 6002would instead be displayed to the right of the second representation ofphoto 6046. Ordering the multiple media item representations in anenvironment in accordance with the time of selection of the respectivemedia items performs an operation (e.g., the operation of determiningthe order of the media item representations in the environment) based ona selection order without requiring further user input to order themultiple media items in the environment. Performing an operationautomatically without requiring further user input enhances theoperability of the device, which, reduces power usage and improvesbattery life of the device by enabling the user to use the device morequickly and efficiently.

In some embodiments, while displaying the first representation of thefirst media item in the second user interface and displaying the secondrepresentation of the second media item in the second user interface,the device detects (1026) an input for manipulating a secondrepresentation of a respective media item (e.g. an input that includes aswipe gesture (for altering a position of the second representation ofthe respective media item on a respective plane in the field of view ofthe one or more cameras or for moving the second representation of therespective media item from a first respective plane in the field of viewof the one or more cameras to a second respective plane in the field ofview of the one or more cameras), an input that includes a de-pinchgesture (for increasing a size of the second representation of therespective media item), an input that includes a pinch gesture (fordecreasing the size of the second representation of the respective mediaitem), and/or an input that includes a rotation gesture (for rotatingthe second representation of the respective media item). In response todetecting the input for manipulating the second representation of therespective media item: in accordance with a determination that the firstinput is directed to the second representation of the first media item(e.g., a swipe input directed to photo 6002 as described with regard toFIGS. 6O-6P), the device manipulates the second representation of thefirst media item based on the first input (e.g., photo 6002 moves inaccordance with the swipe input); and in accordance with a determinationthat the first input is directed to the second representation of thesecond media item (e.g., a swipe input directed to photo 6046 asdescribed with regard to FIGS. 6Q-6R), the device manipulates the secondrepresentation of the second media item based on the first input (e.g.,as photo 6046 moves in accordance with the swipe input). FIGS. 6P-6Yillustrate examples of inputs for manipulating media items 6002, 6046and 6052 in augmented reality user interface 5022. In some embodiments,the input for manipulating a second representation of a respective mediaitem is detected while only a single media item (e.g., the secondrepresentation of the first media item) is displayed and the displayedrepresentation of the media item is manipulated in response to theinput. Manipulating the respective media item representation in theenvironment in response to detecting the input reduced the number ofinputs needed to perform an operation (e.g., by allowing the user todirectly manipulate the displayed objects without requiring additionalinputs to designate an item to be manipulated). Reducing the number ofinputs needed to manipulate the media item representations enhances theoperability of the device, which, additionally, reduces power usage andimproves battery life of the device by enabling the user to use thedevice more quickly and efficiently.

In some embodiments, the sharing user interface includes (1028) a thirduser interface object for transmitting the first media item to a remotedevice (e.g., via an e-mail, a message (e.g., SMS or MMS) or adevice-to-device file transferring service (e.g., AirDrop). For example,as described with regard to FIG. 6C, sharing user interface 6016includes user interface objects (labelled with contact names “John” and“Mary,” as indicated in region 6018) for transmitting photo 6002 to acontact via a device-to-device file transferring service and userinterface objects (labelled “Mail” and “Message” in region 6020) fortransmitting photo 6002 to a remote device via e-mail or via message,respectively. In some embodiments, the sharing user interface includes afourth user interface object for viewing and/or storing the first mediaitem in an application (e.g., an application (such as a notes managementapplication), that is distinct from the application (such as a photomanagement application) that displays the first user interface object).For example, as described with regard to FIG. 6C, sharing user interface6016 includes a user interface object (labelled “Notes” in region 6018)for storing photo 6002 to a notes application that is distinct from thephoto management application. In some embodiments, the sharing userinterface includes a fifth user interface object for copying the firstmedia item. For example, as described with regard to FIG. 6C, sharinguser interface 6016 includes a user interface object (labelled “Copy” inregion 6022) for copying photo 6002. In some embodiments, the sharinguser interface includes sixth user interface object for ceasing todisplay the sharing user interface. For example, as described withregard to FIG. 6C, a user interface object (control 6028 for ceasing todisplay the sharing user interface 6016) is displayed for ceasing todisplay sharing user interface 6016. Including a user interface objectfor transmitting a media item to a remote device on the same sharinguser interface that includes the option for displaying the media item ina field of view of one or more cameras provides additional controloptions without cluttering the user interface (e.g., with controls fordisplaying additional user interfaces). Providing additional controloptions without cluttering the user interface with additional displayedcontrols enhances the operability of the device, which, additionally,reduces power usage and improves battery life of the device by enablingthe user to use the device more quickly and efficiently.

In some embodiments, the second representation of the first media item(e.g., the cutout version of photo 6110) is displayed (1030) with anorientation that is perpendicular to a plane (e.g., a floor or aceiling) in the field of view of the one or more cameras. For example,as described with regard to FIGS. 6AE-6AH, the cutout version of photo6110 is displayed with an orientation that is perpendicular to floorplane 5046 (e.g., such that floor plane 5046 appears to support thecutout version of photo 6110). In some embodiments, a background of thefirst media item is transparent (e.g., an outline of a subject of thefirst media item is detected and a background that includes one or moreportions of the first media item that are beyond the outline (e.g.,portions of photo 6110 that are beyond the cutout version of photo 6110)are removed from the second representation of the first media item suchthat a physical environment as captured in the field of view of the oneor more cameras is visible around the outline and “through” thetransparent portion of the second representation of the first mediaitem. For example, physical table 5020 is visible around the outline ofthe cutout version of the photo 6110. Displaying the media itemrepresentation with an orientation that is perpendicular to a plane inthe field of view of the one or more cameras provides additional controloptions to the user without cluttering the user interface withadditional displayed control (e.g., the user is enabled to display amedia item at a fixed position that is parallel to a vertical surface orperpendicular to a horizontal surface using controls accessed via thesame sharing user interface). Providing additional control options tothe user enhances the operability of the device, which, additionally,reduces power usage and improves battery life of the device by enablingthe user to use the device more quickly and efficiently.

In some embodiments, while a first face of the second representation ofthe first media item is displayed, the device detects (1032) a secondmovement of the electronic device that adjusts the field of view of theone or more cameras (e.g., a movement of device 100 as described withregard to FIGS. 6AE-6AI). In response to detecting the second movementof the electronic device, the device adjusts the second representationof the first media item (e.g., the cutout version of photo 6110) inaccordance with a fixed spatial relationship (e.g., orientation and/orposition) between the second representation of the first media item andthe plane in the field of view of the one or more cameras. After thesecond movement, a second face, opposite the first face, of the secondrepresentation of the first media item is displayed and the second faceof the second representation of the first media item is a reversedversion of the first face of the second representation of the firstmedia item. For example, the face of the cutout version of photo 6110displayed in FIG. 6AI is opposite the face of the cutout version ofphoto 6110 displayed in FIG. 6AE. Displaying a second face, opposite thefirst face, of the second representation of the first media item inresponse to movement of the electronic device provides improved visualfeedback to the user (e.g., indicating a position of the user relativeto the photo), which enhances the operability of the device, which,additionally, reduces power usage and improves battery life of thedevice by enabling the user to use the device more quickly andefficiently.

It should be understood that the particular order in which theoperations in FIGS. 10A-10C have been described is merely an example andis not intended to indicate that the described order is the only orderin which the operations could be performed. One of ordinary skill in theart would recognize various ways to reorder the operations describedherein. Additionally, it should be noted that details of other processesdescribed herein with respect to other methods described herein (e.g.,methods 900, 1100, 1200, and 1400) are also applicable in an analogousmanner to method 1000 described above with respect to FIGS. 10A-10C. Forexample, the contacts, inputs, gestures, media items, environments, userinterface regions, user interface objects, fields of view, and/ormovements described above with reference to method 1000 optionally haveone or more of the characteristics of the contacts, inputs, gestures,media items, virtual objects environments, virtual objects, userinterface regions, user interface objects, fields of view, and/ormovements described herein with reference to other methods describedherein (e.g., methods 900, 1100, 1200, and 1400). For brevity, thesedetails are not repeated here.

FIGS. 11A-11D are flow diagrams illustrating method 1100 for respondingto an input directed to a respective virtual object of a plurality ofdisplayed virtual objects, in accordance with some embodiments. Method1100 is performed at an electronic device (e.g., device 300, FIG. 3, orportable multifunction device 100, FIG. 1A) with a display generationcomponent (e.g., a display, a projector, a heads-up display, etc.), oneor more input devices (e.g., a touch-screen display that serves both asthe display and the touch-sensitive surface, cameras, controllers,joysticks, buttons, etc.), and one or cameras (e.g., one or morerear-facing cameras on a side of the device opposite from the displayand the touch-sensitive surface). In some embodiments, the displaygeneration component is a touch-screen display and the touch-sensitivesurface is an input device that is on or integrated with the displaygeneration component. In some embodiments, the display generationcomponent is separate from one or more input devices. Some operations inmethod 1100 are, optionally, combined and/or the order of someoperations is, optionally, changed.

This method relates to changing the display property of a single virtualobject or multiple virtual objects based on a manipulation input,depending on whether the multiple virtual objects are displayed with aportion of a field of view of one or more cameras. Either changing thedisplay property of a single virtual object or changing the displayproperty of multiple virtual objects, depending on whether the multiplevirtual objects are being displayed with a portion of a field of view ofcameras provides additional control options without cluttering the userinterface with additional displayed controls (e.g., the same input isused to perform different operations rather than requiring separatecontrols for the different operations). Providing additional controloptions without cluttering the user interface with additional displayedcontrols enhances the operability of the device and makes user-deviceinterface more efficient, which, additionally, reduces power usage andimproves battery life of the device by enabling the user to use thedevice more quickly and efficiently.

The device displays (1102), by the display generation component (e.g.,touch-sensitive display system 112), a plurality of virtual objects(e.g., graphical representations of three-dimensional objects). Forexample, as described with regard to FIG. 7A, virtual objects 7002,7004, and 7006 are displayed in object staging user interface 5004; andas described with regard to FIG. 7H, virtual objects 7002, 7004, and7006 are displayed in augmented reality user interface 5022 thatincludes at least a portion of a field of view of the one or morecameras (e.g., one or more cameras of camera module 143).

While displaying the plurality of virtual objects, the device detects(1104) a first manipulation input directed to a respective virtualobject of the plurality of virtual objects (e.g., a pinch or swipe at alocation on the touch-sensitive surface that corresponds to: the firstvirtual object, the second virtual object, another virtual object in theplurality of virtual objects, or a space adjacent to the respectivevirtual object of the plurality of virtual objects). For example, asdescribed with regard to FIG. 7B, an upward swipe input is detected at alocation that corresponds to virtual object 7002 displayed in objectstaging user interface 5004. In FIG. 7H, an upward swipe input isdetected at a location that corresponds to virtual object 7002 displayedin augmented reality user interface 5022.

In response to detecting the first manipulation input directed to therespective virtual object of the plurality of virtual objects (1106): inaccordance with a determination that the plurality of virtual objectsare displayed with at least a portion of a field of view of the one ormore cameras, the device changes a display property of the respectivevirtual object based on the first manipulation input (e.g., individuallyrotating, resizing, and/or moving the respective virtual object), and,in accordance with a determination that the plurality of virtual objectsare displayed in a user interface that does not include the field ofview of the one or more cameras (e.g., a staging user interface thatincludes a simulated three-dimensional space in which athree-dimensional representation of the virtual object may bemanipulated (e.g., rotated around an x-axis, a y-axis, and/or a Z-axis,and/or resized) in response to user input), the device changes a displayproperty of the plurality of virtual objects (e.g., collectivelyrotating and/or resizing the plurality of virtual objects) based on thefirst manipulation input. For example, in response to the inputdescribed with regard to FIG. 7B, while virtual object 7002 is displayedin object staging user interface 5004 that does not include the field ofview of the one or more cameras, a display property of virtual objects7002, 7004, and 7006 is changed (e.g., virtual objects 7002, 7004, and7006 are all simultaneously rotated, as illustrated in FIGS. 7B-7C). Inresponse to the input described with regard to FIG. 7H, while virtualobject 7002 is displayed in object augmented reality user interface 5022that does include the field of view of the one or more cameras, adisplay property of virtual object 7002 is changed (e.g., virtualobjects 7002 is moved, as illustrated in FIGS. 7H-7I, and no othervirtual objects are moved (virtual object 7004 and 7006 remain at thesame position from FIG. 7H to FIG. 7I)).

In some embodiments, while the plurality of virtual objects aredisplayed in the user interface that does not include the field of viewof the one or more cameras, the plurality of virtual objects aredisplayed (1108) with a first set of display characteristics. Displayingthe virtual objects with a first set of display characteristics when thevirtual objects are displayed in a user interface including the field ofview of cameras provides improved visual feedback to the user (e.g., byallowing the user to determine how the virtual objects are displayed).Providing improved visual feedback to the user increases the efficiencywith which the user is able to play with the virtual objects, therebyenhancing the operability of the device, which additionally, reducespower usage and improves battery life of the device by enabling the userto use the device more quickly and efficiently.

In some embodiments, the first set of display characteristics includes(1110) a height ordering characteristic and displaying the plurality ofobjects in accordance with the height ordering characteristic includesordering the respective virtual objects of the plurality of virtualobjects in accordance with respective heights of the respective virtualobjects (e.g., a first virtual object that has a first height isdisplayed to the left of, above, and/or in front of a second virtualobject that has a second height). For example, as illustrated in FIG.7X, virtual objects 7002, 7004, and 7006 are displayed in descendingheight order. In some embodiments, the plurality of objects aredisplayed in ascending height order. Displaying the virtual objects inaccordance with the height ordering characteristic provides improvedvisual feedback to the user (e.g., by allowing the user to readilyunderstand the quantity and distribution of virtual objects). Providingimproved visual feedback to the user enhances the operability of thedevice, which additionally, reduces power usage and improves batterylife of the device by enabling the user to use the device more quicklyand efficiently.

In some embodiments, the first set of display characteristics includes(1112) a spacing characteristic and displaying the plurality of objectsin accordance with the spacing characteristic includes positioning therespective virtual objects of the plurality of virtual objects with auniform distance between respective boundaries of adjacent respectivevirtual objects. For example, as illustrated in FIG. 7X, virtual objects7002, 7004, and 7006 are displayed with a uniform distance between theboundaries (e.g., outlines) of the respective virtual objects.Displaying the virtual objects in accordance with the spacingcharacteristics including positioning of the respective virtual objectswhen the virtual objects are displayed in a user interface including thefield of view of cameras provides improved visual feedback to the user(e.g., by allowing the user to readily understand the quantity anddistribution of virtual objects). Providing improved visual feedback tothe user increases the efficiency with which the user is able to playwith the virtual objects, thereby enhancing the operability of thedevice, which additionally, reduces power usage and improves batterylife of the device by enabling the user to use the device more quicklyand efficiently.

In some embodiments, a first respective virtual object of the pluralityof virtual objects has (1114) a first boundary (e.g., an outline of thefirst respective virtual object or an oval or box that circumscribes thefirst respective virtual object) that corresponds to a static state ofthe first respective virtual object. For example, as described withregard to FIG. 7Z, virtual object 7090 has a first boundary 7096 that isa box that surrounds (e.g., the outer edges of) virtual object 7090. Thefirst respective virtual object is associated with a first action (e.g.,an animation sequence that animates at least a portion of the virtualobject) that causes at least a portion of the first virtual object toextend beyond the first boundary (e.g., the virtual object grows in sizeduring the animation). For example, as described with regard to FIGS.7Z-7AB, virtual object 7090 has an animated sequence that causes virtualobject 7090 to move beyond first boundary 7096. The first respectivevirtual object has a second boundary that corresponds to the extendedrange of the first respective virtual object during the first action(e.g., an outline of the first respective virtual object thatincorporates the extension caused by the first action or an oval or boxthat circumscribes the first respective virtual object including theextension caused by the first action). For example, as described withregard to FIGS. 7AB-7AC, second boundary 7108 of virtual object 7090 toan extended range of virtual object 7090 including the movement ofvirtual object 7090 beyond boundary 7096 during the animated sequence.The respective boundaries of the adjacent respective virtual objectsinclude the second boundary of the first respective virtual object. Forexample, virtual object 7088 has boundary 7094 that is a box thatsurrounds (e.g., the outer edges of) virtual object 7088, virtual object7090 has second boundary 7108 of virtual object 7090 to an extendedrange of virtual object 7090, and virtual object 7092 has boundary 7098that is a box that surrounds (e.g., the outer edges of) virtual object7092. Virtual objects 7088, 7090, and 7092 are displayed with a uniformdistance between the boundaries (e.g., outlines) of the respectivevirtual objects. The virtual object having a first boundary thatcorresponds to the static state of the virtual object and a secondboundary that corresponds to the extended range of the virtual objectduring the associated action provides improved visual feedback to theuser (e.g., by allowing the user to determine the extent of animation ofthe virtual object and manipulate the virtual object in accordance withthe information about the extent of the animation). Providing improvedvisual feedback to the user increases the efficiency with which the useris able to interact with the virtual object (e.g. by helping the user toprovide proper inputs and reducing user mistakes whenoperating/interacting with the device), thereby enhancing theoperability of the device, which, additionally, reduces power usage andimproves battery life of the device by enabling the user to use thedevice more quickly and efficiently.

In some embodiments, while displaying the plurality of virtual objectswith at least a portion of a field of view of the one or more cameras(e.g., in augmented reality user interface 5022), after changing thedisplay property of the respective virtual object based on the firstmanipulation input, the device detects (1116) a request to display theplurality of virtual objects in the user interface that does not includethe field of view of the one or more cameras (e.g., object staging userinterface 5004). For example, after display properties of virtualobjects 7002, 7004, and 7006 have been changed in response to inputsprovided while augmented reality user interface 5022 is displayed, asdescribed with regard to FIGS. 7H-7R, an input at a location thatcorresponds to the “Object” region of toggle control 5008 is provided asa request to display virtual objects 7002, 7004 and 7006 in objectstaging interface 5004, as described with regard to FIG. 7U. In responseto detecting the request to display the plurality of virtual objects inthe user interface that does not include the field of view of the one ormore cameras, the device displays the user interface that does notinclude the field of view of the one or more cameras, includingdisplaying the plurality of virtual objects with the first set ofdisplay characteristics. For example, as described with regard to FIGS.7U-7X, virtual objects 7002, 7004, and 7006 are displayed in objectstaging interface 5004 with the same positions, orientations, and orderthat virtual objects 7002, 7004, and 7006 had in object staginginterface 5004 prior to the changes to the display properties thatoccurred in response to inputs provided while augmented reality userinterface 5022 was displayed. For example, the positions, orientations,sizes, and order of virtual objects 7002, 7004, and 7006 in FIG. 7X arethe same as the positions, orientations, sizes, and order of virtualobjects 7002, 7004, and 7006 in FIG. 7A. Displaying the virtual objectswith the first set of display characteristics when displaying the userinterface that does not include the field of view of the one or morecameras reduces the number of inputs needed to perform an operation(e.g., by displaying all of the virtual objects in readily accessiblepositions, such that the user does not need to reposition a virtualobjects in order to view and manipulate it). Reducing the number ofinputs needed to perform an operation increases the efficiency withwhich the user is able to switch between different display, therebyenhancing the operability of the device, which, additionally, reducespower usage and improves battery life of the device by enabling the userto use the device more quickly and efficiently.

In some embodiments, while displaying the plurality of virtual objectsin the user interface that does not include the field of view of the oneor more cameras (e.g., object staging user interface 5004), the devicedetects (1118) a request to re-display the plurality of virtual objectswith at least a portion of a field of view of the one or more cameras(e.g., in augmented reality user interface 5022). For example, afterdisplay properties of virtual objects 7002, 7004, and 7006 have beenchanged in response to inputs provided while augmented reality userinterface 5022 is displayed, as described with regard to FIGS. 7H-7R,and after virtual objects 7002, 7004 and 7006 are displayed in objectstaging interface 5004, as described with regard to FIG. 7X, an input ata location that corresponds to the “AR” region of toggle control 5008 isprovided as a request to re-display virtual objects 7002, 7004 and 7006in augmented reality user interface 5022, as described with regard toFIG. 7X. In response to detecting the request to re-display theplurality of virtual objects with at least a portion of a field of viewof the one or more cameras, the device re-displays the plurality ofvirtual objects with at least a portion of a field of view of the one ormore cameras, wherein the display property of the respective virtualobject is changed based on the first manipulation input. For example, asdescribed with regard to FIGS. 7X-7Y, virtual objects 7002, 7004 and7006 are re-displayed in augmented reality user interface 5022. In FIG.7Y, virtual objects 7002, 7004, and 7006 are displayed in accordancewith the changes to the positions, orientations, sizes, and order thatwere made in response to inputs provided, as described with regard toFIGS. 7H-7R. Re-displaying the respective virtual object with a displayproperty based on the first manipulation input reduces the number ofinputs needed (e.g., by re-applying the display property to therespective virtual object without requiring the user to provide theinput to manipulate the respective virtual object a second time).Reducing the number of inputs needed to perform an operation increasesthe efficiency with which the user is able to interact with virtualobjects, thereby enhancing the operability of the device, which,additionally, reduces power usage and improves battery life of thedevice by enabling the user to use the device more quickly andefficiently.

In some embodiments, the first manipulation input is a rotation inputand changing the display property of the plurality of virtual objects(in the user interface that does not include the field of view of theone or more cameras) includes (1120) rotating the plurality of virtualobjects around a central region (e.g., centroid) of a boundary thatcorresponds to (e.g., surrounds) the plurality of virtual objects. Forexample, as described with regard to FIGS. 7B-7C, virtual objects 7002,7004, and 7006 rotate around a centerline 7014 that is a central regionof boundary 7012. In some embodiments, the boundary that corresponds tothe plurality of virtual objects is an oval or box that circumscribesthe plurality of virtual objects. In some embodiments, at least onevirtual object of the plurality of virtual objects is associated with afirst action (e.g., an animation sequence that animates at least aportion of the virtual object) that causes at least a portion of thefirst virtual object to extend beyond a first boundary that correspondsto the static state of the virtual object (e.g., the virtual objectgrows in size during the animation) and the boundary that corresponds tothe plurality of virtual objects encompasses a second boundary thatcorresponds to the extended range of the at least one virtual object.For example, as described with regard to FIG. 7AC, first boundary 7096corresponds to a static state of virtual object 7090 and second boundary7108 corresponds to an extended range of virtual object 7090. Rotatingthe plurality of virtual objects around a central region of a boundarythat corresponds to the plurality of virtual objects reduces the numberof inputs needed to perform an operation (e.g., by rotating the virtualobjects collectively in response to a single input rather than requiringseveral individual inputs). Reducing the number of inputs needed toperform an operation increases the efficiency with which the user isable to perform these operations, thereby enhancing the operability ofthe device, which, additionally, reduces power usage and improvesbattery life of the device by enabling the user to use the device morequickly and efficiently.

In some embodiments, prior to displaying the plurality of virtualobjects, the device displays (1122) a selection user interface (e.g., aphoto user interface that displays a plurality of selectable images, afile user interface that displays a plurality of selectable files, or awebsite that includes a plurality of selectable objects). While theselection user interface is displayed, the device detects selection ofthe plurality of virtual objects. For example, as described with regardto FIGS. 7AD-7AH, while object management user interface 7099 isdisplayed, selection of virtual objects 7110, 7118, and 7122 isdetected. While the plurality of virtual objects are selected, thedevice receives a request to display the plurality of virtual objects.For example, the request to display the plurality of virtual objectsincludes an input at a location that corresponds to share control 7130,as described with regard to FIG. 7AH, and an input at a location thatcorresponds to control 6024 for displaying one or more selected objectsin an object three-dimensional viewing mode, as described with regard toFIG. 7AI. In response to receiving the request to display the pluralityof virtual objects, the device displays the plurality of virtual objectsin the user interface that does not include the field of view of the oneor more cameras. For example in response to the inputs described withregard to FIGS. 7AH-7AI, virtual objects 7110, 7118, and 7122 aredisplayed in object staging view 5004, as described with regard to FIG.7AJ. In some embodiments, the selection user interface enables selectionof a plurality of virtual objects using a single selection input (e.g.,the selection user interface includes a link for accessing a predefinedgroup of virtual objects). In some embodiments, an anchor plane thatcorresponds to the respective plane is a property in a set of propertiesof a respective virtual object of the plurality of virtual objects, andis specified in accordance with the nature of a physical object that therespective virtual object is supposed to represent. In some embodiments,the virtual object is placed at a predefined orientation and/or positionrelative to multiple planes detected in the field of view of the one ormore cameras (e.g., multiple respective sides of the virtual object areassociated with respective planes detected in the field of view of thecamera(s). In some embodiments, if the orientation and/or positionpredefined for the virtual object is defined relative to a horizontalbottom plane of the virtual object, the bottom plane of the virtualobject is displayed on a floor plane detected in the field of view ofthe camera(s) (e.g., the horizontal bottom plane of the virtual objectis parallel to the floor plane with zero distance from the floor plane).In some embodiments, if the orientation and/or position predefined forthe virtual object is defined relative to a vertical back plane of thevirtual object, the back surface of the virtual object is placed againsta wall plane detected in the field of view of the one or more cameras(e.g., the vertical back plane of the virtual object is parallel to thewall plane with zero distance from the wall plane). In some embodiments,the virtual object is placed at a fixed distance relative a respectiveplane and/or at an angle other than zero or right angles relative to therespective plane. Enabling selection of a plurality of virtual objectsfrom the selection user interface reduces the number of inputs requiredto select multiple objects (e.g., by allowing a user to select multipleobjects from a single user interface rather than requiring the user toseparately select objects from multiple user interfaces). Reducing thenumber of inputs required to perform an operation enhances theoperability of the device, which, additionally, reduces power usage andimproves battery life of the device by enabling the user to use thedevice more quickly and efficiently.

In some embodiments, a first virtual object of the plurality of virtualobjects has a first defined plane type (e.g., the first virtual objectis configured to be displayed at a fixed position relative to a firsttype of plane, such as a vertical (e.g., wall) plane) and a secondvirtual object of the plurality of virtual objects has a second definedplane type (e.g., the second virtual object is configured to bedisplayed at a fixed position relative to a second type of plane, suchas a horizontal (e.g., floor, ceiling, or table surface) plane). Forexample, the defined plane type of virtual object 7118 is a verticalplane and the defined plane type of virtual object 7110 is a horizontalplane. In response to receiving the request to display the plurality ofvirtual objects (1124): the first virtual object of the plurality ofvirtual objects is displayed at a fixed position relative to a firstplane that corresponds to the first defined plane type (e.g., the firstvirtual object is displayed at a fixed position relative to a verticalplane) and the second virtual object of the plurality of virtual objectsis displayed at a fixed position relative to a second plane thatcorresponds to the second defined plane type that is distinct from thefirst defined plane type (e.g., the second virtual object is displayedat a fixed position relative to a horizontal plane). For example, asdescribed with regard to FIG. 7AK, virtual object 7118 is displayed at afixed position relative to vertical plane 7132 and virtual object 7110is displayed at a fixed position relative to horizontal plane 5046.Displaying the first virtual object at a fixed position relative to afirst plane and displaying the second virtual object at a fixed positionrelative to a second plane reduces the number of inputs required toplace multiple virtual objects (e.g., by reducing the number of inputsrequired to individually place multiple virtual objects at positionsrelative to different planes). Reducing the number of inputs required toplace multiple virtual objects enhances the operability of the device,which, additionally, reduces power usage and improves battery life ofthe device by enabling the user to use the device more quickly andefficiently.

In some embodiments, while the first virtual object is displayed at thefixed position relative to the first plane and the second virtual objectis displayed at the fixed position relative to the second plane, thedevice detects (1126) a rotation input. In response to detecting therotation input: in accordance with a determination that the rotationinput is directed to the first virtual object, the device rotates thefirst virtual object about a first axis (e.g., rotating the firstvirtual object about an axis that is normal to a vertical plane relativeto which the first virtual object is displayed) and in accordance with adetermination that the rotation input is directed to the second virtualobject, the device rotates the second virtual object about a second axisthat is distinct from the first axis (e.g., rotating the second virtualobject about an axis that is normal to a horizontal plane relative towhich the second virtual object is displayed). For example, as describedwith regard to FIGS. 7AM-7AO, in response to a rotation input directedto virtual object 7118, virtual object 7118 rotates about an axis thatis normal to vertical plane 7132 and, as described with regard to FIGS.7AO-7AQ, in response to a rotation input directed to virtual object7110, virtual object 7110 rotates about an axis that is normal tohorizontal plane 5046. Rotating the first virtual object and the secondvirtual object about different axes in response to detecting a rotationinput provides reduces the number of inputs needed to perform anoperation (e.g., by automatically determining the appropriate axis ofrotation based on the surface to which the virtual object is affixedwithout requiring the user to provide input to specify an axis ofrotation). Reducing the number of inputs needed to perform rotation ofvirtual objects about different axes in a user interface increases theefficiency with which the user is able to perform these operations,thereby enhancing the operability of the device, which, additionally,reduces power usage and improves battery life of the device by enablingthe user to use the device more quickly and efficiently.

It should be understood that the particular order in which theoperations in FIGS. 11A-11D have been described is merely an example andis not intended to indicate that the described order is the only orderin which the operations could be performed. One of ordinary skill in theart would recognize various ways to reorder the operations describedherein. Additionally, it should be noted that details of other processesdescribed herein with respect to other methods described herein (e.g.,methods 900, 1000, 1200, and 1400) are also applicable in an analogousmanner to method 1100 described above with respect to FIGS. 11A-11D. Forexample, the contacts, inputs, gestures, virtual objects, environments,user interface regions, user interface objects, fields of view,movements, and/or animations described above with reference to method1100 optionally have one or more of the characteristics of the contacts,inputs, gestures, media items, virtual objects, environments, userinterface regions, user interface objects, fields of view, movements,and/or animations described herein with reference to other methodsdescribed herein (e.g., methods 900, 1000, 1200, and 1400). For brevity,these details are not repeated here.

FIGS. 12A-12B are flow diagrams illustrating method 1200 for displayinga prompt to change a property of a media item that does not meetcompatibility criteria for display in an augmented reality environment,in accordance with some embodiments. Method 1200 is performed at anelectronic device (e.g., device 300, FIG. 3, or portable multifunctiondevice 100, FIG. 1A) with a display generation component (e.g., adisplay, a projector, a heads-up display, etc.), one or more inputdevices (e.g., a touch-screen display that serves both as the displayand the touch-sensitive surface, cameras, controllers, joysticks,buttons, etc.), and one or cameras (e.g., one or more rear-facingcameras on a side of the device opposite from the display and thetouch-sensitive surface). In some embodiments, the display generationcomponent is a touch-screen display and the touch-sensitive surface isan input device that is on or integrated with the display generationcomponent. In some embodiments, the display generation component isseparate from one or more input devices. Some operations in method 1200are, optionally, combined and/or the order of some operations is,optionally, changed.

The device displays (1202), by the display generation component (e.g.,touch-sensitive display system 112), a representation of a media item(e.g., a three-dimensional model, a photograph, an image, a still imagethat corresponds to a video, album art that corresponds to musiccontent, or a thumbnail representation thereof). For example, in FIG.8A, AR-incompatible media item 8002 is displayed in file management userinterface 8000. In FIG. 8G, AR-compatible media item 8022 is displayedin file management user interface 8000.

The device detects (1204) an input corresponding to a request to displaythe media item in an augmented reality environment. For example, theinput includes input detected at a location that corresponds to sharecontrol 7130 as described with regard to FIG. 8B and/or FIG. 8H and/orinput detected at a location that corresponds to control 6024, asdescribed with regard to FIG. 8C and/or FIG. 8I.

In response to detecting the input corresponding to a request to displaythe media item in an augmented reality environment (1206): in accordancewith a determination that the media item has a property (e.g., mediaformat, data size, or corresponding physical size) that does not meetcompatibility criteria for display in the augmented reality environment(e.g., the incompatible media format is wavefront, collada, PLY,Alembic, gITF, or dae), the device displays a prompt (e.g., replacingdisplay of the representation of the media item and/or overlaid overdisplay of the media item) to change the property of the media item(e.g., convert the media item from a first format to a compatibleformat, apply compression, or adjust a size property of the object) and,in accordance with a determination that the media item meets thecompatibility criteria (e.g., the compatible media format is a format ofa three-dimensional virtual object or the media format is compatiblewith an augmented reality platform, such as the AR Kit platform), thedevice displays a virtual object that corresponds to the media item withcontent of at least a portion of the field of view of the one or morecameras. For example, media item 8002 has a property that does not meetcompatibility criteria for display in an augmented reality environment,so in response to the request to display media item 8002 in an augmentedreality environment, prompt 8012 to change the property of the mediaitem is displayed. Media item 8022 meets compatibility criteria fordisplay in an augmented reality environment, so in response to therequest to display media item 8022 in an augmented reality environment,virtual object 8023 that corresponds to media item 8022 is displayed inaugmented reality environment 5022, as illustrated in FIG. 8K.Displaying a virtual object corresponding to the media item that meetsthe compatibility criteria or displaying a prompt to change the propertyof the media item that does not meet the compatibility criteria, inresponse to detecting the input corresponding to a request to displaythe media item in an augmented reality environment, provides improvedvisual feedback to the user (e.g., by notifying the user whether themedia item meets compatibility criteria). Providing improved visualfeedback to the user increases the efficiency with which the user isable to display the media item as the virtual object, thereby enhancingthe operability of the device, which, additionally, reduces power usageand improves battery life of the device by enabling the user to use thedevice more quickly and efficiently.

In some embodiments, while displaying the representation of the mediaitem, the device displays (1208) a control (e.g., a button thatindicates an option for viewing the media item in AR) that correspondsto the media item and the input corresponding to the request to displaythe media item in the augmented reality environment is an input detectedat a location that corresponds to the control that corresponds to themedia item. Displaying a control that corresponds to the media item anddetecting an input at the location that corresponds to the controlprovides additional control options without cluttering the UI withadditional displayed controls. Providing additional control optionswithout cluttering the UI with additional displayed controls enhancesthe operability of the device, which, additionally, reduces power usageand improves battery life of the device by enabling the user to use thedevice more quickly and efficiently.

In some embodiments, the one or more input devices include atouch-screen display and the device detects (1210) an input by a contactwith the touch-screen at a location that corresponds to therepresentation of the media item. In response to detecting the input bythe contact with the touch-screen at the location that corresponds tothe representation of the media item, in accordance with a determinationthat the input meets option display criteria, the device displays anoption for displaying the media item in the augmented realityenvironment. The input corresponding to the request to display the mediaitem in the augmented reality environment is an input detected at alocation that corresponds to the option for displaying the media item inthe augmented reality environment. Displaying an option for displayingthe media item in the augmented reality environment in accordance with adetermination that the input meets option display criteria providesadditional control options without cluttering the UI with additionaldisplayed controls. Providing additional control options withoutcluttering the UI with additional displayed controls enhances theoperability of the device, which, additionally, reduces power usage andimproves battery life of the device by enabling the user to use thedevice more quickly and efficiently.

In some embodiments, the option display criteria include (1212) criteriathat are satisfied in accordance with a determination that the contactis maintained at the location on the touch-sensitive surface thatcorresponds to the representation of the media item with less than athreshold amount of movement for at least a predefined threshold amountof time (e.g., a long press time threshold). For example, the firstcriteria are met by a touch-hold input.

In some embodiments, the compatibility criteria include criteria thatare satisfied in accordance with a determination that (1214) a fileformat of the media item is a file format (e.g., the compatible mediaformat is a format of a three-dimensional virtual object or the mediaformat is compatible with an augmented reality platform, such as the ARKit platform) that is compatible with display in the augmented realityenvironment. For example, the file format is USDZ.

In some embodiments, the compatibility criteria include criteria thatare satisfied in accordance with a determination that (1216) a data sizeof the media item meets size criteria (e.g., the data size is below adefined threshold).

In some embodiments, the media item includes (1218) metadata thatindicates a physical size of that corresponds to the media item (and, insome embodiments, a plane type that corresponds to the media item (e.g.,wall, floor, ceiling)), the device detects, in the field of view of theone or more cameras, at least one plane; and the compatibility criteriainclude criteria that are satisfied in accordance with a determinationthat a physical size of the media item meets placement criteria for theat least one plane (e.g., the physical size of the media item is notlarger than a determined size of the plane). In some embodiments, theplacement criteria require that a plane type that corresponds to aspecified plane type of the media item is detected in the field of viewof the one or more cameras and/or that the physical size of the mediaitem is not larger than a detected plane of the plane type. Satisfyingthe compatibility criteria in accordance with a determination that thephysical size of the media item meets placement criteria for the planereduces the number of inputs needed to perform an operation (e.g.,allowing the user to determine whether the media item meets thecompatibility criteria for displaying a virtual object in the augmentedreality environment). Reducing the number of inputs needed to perform anoperation enhances the operability of the device, which, additionally,reduces power usage and improves battery life of the device by enablingthe user to use the device more quickly and efficiently.

In some embodiments, while displaying the prompt to change the propertyof the media item, the device detects (1220) an input for accepting thesuggestion to change the property of the media item (e.g., an input at alocation that corresponds to control 8014 for converting the selectedmedia item to an AR-compatible format, as described with regard to FIG.8D) and, in response to detecting the input for accepting the suggestionto change the property of the media item, the device changes theproperty of the media item (e.g., the device converts the media itemfrom a first format to a compatible format, apply compression, or adjusta size property of the object). For example, in response to detectingthe input for accepting the suggestion to change the property of themedia item, media item 8002 is converted to virtual object 8003.Changing the property of the media item in response to detecting theinput for accepting the suggestion to change the property of the mediaitem reduces the number of inputs required to change the property of themedia item (e.g., by changing the property without requiring the user toprovide input to exit the application and use an external application tochange the property of the media item). Reducing the number of inputsrequired to change the property of the media item enhances theoperability of the device, which, additionally, reduces power usage andimproves battery life of the device by enabling the user to use thedevice more quickly and efficiently.

In some embodiments, in response to detecting the input for acceptingthe suggestion to change the property of the media item, the devicedisplays (1222) the virtual object that corresponds to the media itemwith content of at least a portion of the field of view of the one ormore cameras (e.g., virtual object 8003 corresponding to the convertedversion of media item 8002 is displayed in augmented reality userinterface 5022, as described with regard to FIG. 8F). Displaying thevirtual object that corresponds to the media item in a user interfacethat includes content of at least a portion of the field of view ofcameras in response to detecting the input for accepting the suggestionto change the property of the media item reduces the number of inputsrequired to change the property of the media item (e.g., by changing theproperty without requiring the user to provide input to exit theapplication and use an external application to change the property ofthe media item). Reducing the number of inputs required to change theproperty of the media item enhances the operability of the device,which, additionally, reduces power usage and improves battery life ofthe device by enabling the user to use the device more quickly andefficiently.

In some embodiments, in response to detecting the input for acceptingthe suggestion to change the property of the media item, the devicedisplays (1224) the virtual object that corresponds to the media item ina user interface that does not include the field of view of the one ormore cameras (e.g., a staging user interface that includes a simulatedthree-dimensional space in which a three-dimensional representation ofthe virtual object may be manipulated (e.g., rotated around an x-axis, ay-axis, and/or a Z-axis, and/or resized) in response to user input). Forexample, virtual object 8003 corresponding to the converted version ofmedia item 8002 is displayed in object staging user interface 5004, asdescribed with regard to FIG. 8E. Displaying the virtual object thatcorresponds to the media item in a user interface that does not includescontent of at least a portion of the field of view of cameras inresponse to detecting the input for accepting the suggestion to changethe property of the media item reduces the number of inputs required tochange the property of the media item (e.g., by changing the propertywithout requiring the user to provide input to exit the application anduse an external application to change the property of the media item).Reducing the number of inputs required to change the property of themedia item enhances the operability of the device, which, additionally,reduces power usage and improves battery life of the device by enablingthe user to use the device more quickly and efficiently.

In some embodiments, while displaying the prompt to change the propertyof the media item, the device detects (1226) an input for rejecting thesuggestion to change the property of the media item and, in response todetecting the input for rejecting the suggestion to change the propertyof the media item, the device displays an error message instead ofdisplaying the virtual object in a user interface that includes thefield of view of the one or more cameras. Displaying an error message inresponse to detecting the input for rejecting the suggestion to changethe property of the media item provides improved visual feedback to theuser (e.g., indicating to the user that the suggestion to change theproperty of the media item has been rejected). Providing improved visualfeedback enhances the operability of the device, which, additionally,reduces power usage and improves battery life of the device by enablingthe user to use the device more quickly and efficiently.

It should be understood that the particular order in which theoperations in FIGS. 12A-12B have been described is merely an example andis not intended to indicate that the described order is the only orderin which the operations could be performed. One of ordinary skill in theart would recognize various ways to reorder the operations describedherein. Additionally, it should be noted that details of other processesdescribed herein with respect to other methods described herein (e.g.,methods 900, 1000, 1100, and 1400) are also applicable in an analogousmanner to method 1200 described above with respect to FIGS. 12A-12B. Forexample, the contacts, inputs, gestures, media items, virtual objects,environments, user interface regions, user interface objects, and/orfields of view described above with reference to method 1200 optionallyhave one or more of the characteristics of the contacts, inputs,gestures, media items, virtual objects, environments, user interfaceregions, user interface objects, fields of view, movements, and/oranimations described herein with reference to other methods describedherein (e.g., methods 900, 1000, 1100, and 1400). For brevity, thesedetails are not repeated here.

FIGS. 13A-13J illustrate example user interfaces for displaying avirtual model of content concurrently with a selectable user interfaceobject for performing an operation associated with the content, inaccordance with some embodiments. The user interfaces in these figuresare used to illustrate the processes described below, including theprocesses in FIGS. 9A-9C, 10A-10C, 11A-11D, 12A-12B, and 13A-13B. Forconvenience of explanation, some of the embodiments will be discussedwith reference to operations performed on a device with atouch-sensitive display system 112. In such embodiments, the focusselector is, optionally: a respective finger or stylus contact, arepresentative point corresponding to a finger or stylus contact (e.g.,a centroid of a respective contact or a point associated with arespective contact), or a centroid of two or more contacts detected onthe touch-sensitive display system 112. However, analogous operationsare, optionally, performed on a device with a display 450 and a separatetouch-sensitive surface 451 in response to detecting the contacts on thetouch-sensitive surface 451 while displaying the user interfaces shownin the figures on the display 450, along with a focus selector.

FIGS. 13A-13G are related to a shopping user interface 1300 that isconfigured to perform an operation (e.g., a purchase operation)associated with content.

FIG. 13A illustrates shopping user interface 1300 that displays contentthat includes content item 1302 (a chair), content item 1304 (a bedframe), content item 1306 (a lamp), and content item 1308 (a couch).Virtual models of content items 1302, 1304, and 1308 are available(e.g., for display in an object view and/or an augmented reality view asdescribed further below with regard to FIGS. 13B-13C), as indicated byvirtual model indicators 1310, 1312, and 1314, respectively. Forexample, virtual model indicator 1310 displayed adjacent to content item1302 indicates that a virtual model of content item 1302 is available toview. Because no virtual model indicator is displayed for content item1306, the user is made aware that no virtual model is available forcontent item 1306. The shopping user interface 1300 includes metadata1316, 1318, 1320, and 1322 that corresponds to content items 1302, 1304,1306, and 1308, respectively. For example, metadata 1316 includes a name(“Modern Chair”) of content item 1302, a material (“Material: Wood”) ofcontent item 1302, a price (“Price: $499”) of content item 1302,additional information (“Free Shipping!”) for content item 1302, and/ora rating (three out of five stars) for content item 1302.

While shopping user interface 1300 is displayed, a request to display avirtual model that corresponds to content item 1308 is received. Forexample, an input (e.g., a tap input) by a contact 1324 is detected at alocation that corresponds to content item 1308 (e.g., at a location thatcorresponds to the displayed representation of the content item and/orat a location that corresponds to metadata associated with the contentitem), as illustrated in FIG. 13A. In some embodiments, in response tothe request to display the virtual model that corresponds to a contentitem, the virtual model is displayed in an object staging user interface(e.g., object staging user interface 5004, as illustrated in FIG. 13B).In some embodiments, in response to the request to display the virtualmodel that corresponds to a content item, the virtual model is displayedin an augmented reality user interface (e.g., augmented reality userinterface 5022, as illustrated in FIG. 13C).

As illustrated in FIG. 13B, in response to the input by contact 1324, avirtual model 1326 of content item 1308 is displayed concurrently withselectable user interface object 1328 (e.g., a purchase button) inobject staging user interface 5004. While virtual model 1326 isdisplayed in object staging user interface 5004, user input formanipulation of virtual model 1326 causes a characteristic of virtualmodel 1326 to change (e.g. rotation of the virtual object about one ormore axes). For example, manipulation of virtual model 1326 in objectstaging user interface 5004 occurs as described with regard to FIGS.5B-5C above. In some embodiments, at least a portion of the metadata1322 for content item 1308 is displayed in object staging user interface5004, as indicated at 1330.

While virtual model 1326 is displayed in object staging user interface5004, as shown in FIG. 13B, an input (e.g., a tap input) by contact 1332is detected on touch-sensitive display system 112 at a location thatcorresponds to the “AR” region of toggle control 5008. In response tothe input, display of object staging user interface 5004 is replaced bydisplay of virtual model 1326 in an augmented reality user interface5022 (e.g., virtual model 1326 is displayed in a physical environmentcaptured by one or more cameras of device 100), as illustrated in FIG.13C.

In FIG. 13C, virtual model 1326 of content item 1308 is displayedconcurrently with selectable user interface object 1328 in augmentedreality user interface 5022. In some embodiments, virtual model 1326 isdisplayed at a location that corresponds to a surface (e.g., a floorsurface) detected in the physical environment captured by one or morecameras of device 100. Physical table 5020 is visible in the physicalenvironment captured by the one or more cameras. Selectable userinterface object 1328 is translucent (e.g., such that the floor in thephysical environment is partially visible through the “Purchase”button.) In some embodiments, at least a portion of the metadata 1322for content item 1308 is also displayed in augmented reality userinterface 5022, as indicated at 1334.

While virtual model 1326 is displayed concurrently with selectable userinterface object 1328 (e.g., in object staging user interface 5004, asillustrated in FIG. 13B or in augmented reality user interface 5022, asillustrated in FIG. 13C) an input (e.g., a tap input by contact 1336,illustrated in FIG. 13C) is detected at a location that corresponds toselectable user interface object 1328. In some embodiments, in responseto detecting the input at a location that corresponds to selectable userinterface object 1328, a native payment interface 1340 (e.g., a paymentinterface associated with the operating system of device 100, such as anApple Pay interface) is displayed, as shown in FIG. 13D. In someembodiments, in response to detecting the input at a location thatcorresponds to selectable user interface object 1328, a store paymentinterface 1342 (e.g., a payment interface associated with the store thatprovides shopping user interface 1300) is displayed, as shown in FIG.13E.

As illustrated in FIG. 13D, native payment interface 1340 includespayment information (e.g., previously provided credit card information),shipping information (e.g., a previously provided shipping addressand/or shipping method), and contact information for the purchaser(e.g., previously provided contact information). While the nativepayment interface 1340 is displayed, an input to authorize acquisitionof content item 1308 is provided (e.g., the user provides input (e.g., atap input) at a user interface object for authorizing acquisition of thecontent item, the user provides a numeric input at a displayed numberpad, or the user provides biometric information such as a fingerprint orthumbprint at a location that corresponds to a fingerprint sensor and/ora facial recognition input). For example, the input to authorizeacquisition of content item 1308 is a thumbprint input detected by athumbprint sensor at a location indicated by region 1341.

FIG. 13E illustrates store payment interface 1342. In some embodiments,store payment interface includes input fields that prompt the user toprovide input indicating payment information, shipping information,and/or contact information. While the store payment interface 1342 isdisplayed, an input to authorize acquisition of content item 1308 isprovided (e.g., the user provides input (e.g., a tap input) at alocation that corresponds to “Place Order” button 1344).

FIG. 13F illustrates input that corresponds to a request to display avirtual model that includes multiple virtual objects that correspond tomultiple content items. As shown in FIG. 13F, content items 1302, 1304and 1308 have been added to a shopping cart (as indicated by in-cartindicators 1346, 1348, and 1350). For example, content item 1302 hasbeen added to the shopping cart by input at a location that correspondsto a respective add to cart button 1351 that corresponds to content item1302. An input (e.g., a tap input) by a contact 1354 is detected at alocation that corresponds to a user interface object for viewing avirtual model of the selected content items (e.g., “View Cart in AR”link 1353). In some embodiments, the user interface object for viewingthe virtual model of the selected content items is distinct from a userinterface object (e.g., “View Cart” link 1355) for viewing the selectedcontent items (e.g., without displaying virtual objects that correspondto the respective selected content items). For example, from shoppinguser interface 1300, a user is enabled to add individual items to a cart(e.g., using add to cart button 1351), purchase individual items using aselectable user interface object (e.g., purchase button 1352), view avirtual model that corresponds to a content item (e.g., by providinginput at a location that corresponds to the content item), view avirtual model that corresponds to content in the cart (e.g., multiplecontent items in the cart) in an augmented reality user interface 5022(e.g., using “View Cart in AR” link 1353), view the content in the cartwithout displaying a virtual model of the content (e.g., using “ViewCart” link 1355), and/or provide input to proceed directly to a paymentinterface, such as native payment user interface 1340 or store paymentinterface 1342 (e.g., using the “Check Out” link 1357). In response tothe input detected at the location that corresponds to the userinterface object for viewing the virtual model of the selected contentitems, the virtual model is displayed (e.g., in an object staging userinterface 5004 or an augmented reality user interface 5022).

In FIG. 13G, in response to the input detected at the “View Cart in AR”link 1353, display of shopping user interface 1300 has been replaced bydisplay of augmented reality user interface 5022. A virtual model thatincludes virtual object 1356 associated with content item 1302 (thechair), virtual object 1358 associated with content item 1304 (the bed),and virtual object 1360 associated with content item 1308 (the couch) isdisplayed concurrently with selectable user interface object 1328 inaugmented reality user interface 5022. In some embodiments, in responseto the input detected at the “View Cart in AR” link 1353, display ofshopping user interface 1300 is replaced by display of the virtual modelthat includes virtual object 1356, 1358, and 1360 concurrently withselectable user interface object 1328 in object staging user interface5004. In some embodiments, responses to input directed to a respectivevirtual object of a plurality of displayed virtual objects in thevirtual model occur as described with regard to FIGS. 7A-7AQ and11A-11D. For example, in response to a manipulation input directed to arespective virtual object displayed in augmented reality user interface5022, the device changes a display property of the respective virtualobject based on the manipulation input, and in response to amanipulation input directed to a respective virtual object displayed inobject staging user interface 5004, the device changes a displayproperty of the plurality of virtual objects based on the manipulationinput.

FIGS. 13H-13J are related to an object selection user interface 1360that is not configured to perform the purchase operation (e.g., forpurchasing the content items as described with regard to FIGS. 13A-13G).

As illustrated in FIG. 13H, object selection user interface 1360 is afile user interface that displays two-dimensional images that correspondto virtual objects available for viewing. For example, in the objectselection user interface illustrated in FIG. 13H, image 1362 is atwo-dimensional representation of a virtual model 1326. For example,before purchasing content item 1308 using shopping user interface 1300,a user stored virtual model 1362 corresponding to content item 1308 in apersonal collection of virtual models illustrated in object selectionuser interface 1360. While the object selection user interface 1360 isdisplayed, a request to display the virtual model that corresponds tothe content represented by image 1362. For example, an input (e.g., atap input) by contact 1364 is detected at a location that corresponds toimage 1362. In some embodiments, in response to the request to displaythe virtual model that corresponds to the content item, the virtualmodel is displayed in an object staging user interface (e.g., objectstaging user interface 5004, as illustrated in FIG. 13I). In someembodiments, in response to the request to display the virtual modelthat corresponds to the content item, the virtual model is displayed inan augmented reality user interface (e.g., augmented reality userinterface 5022, as illustrated in FIG. 13J).

As illustrated in FIG. 13I, in response to the input by contact 1364,virtual model 1326 of content item 1308 is displayed in object staginguser interface 5004. Because the input to display virtual model 1326 wasreceived in object selection user interface 1360, which is not ashopping user interface for acquiring content, no purchase button(selectable user interface object 1328) is displayed.

As illustrated in FIG. 13I, in response to the input by contact 1364,virtual model 1326 of content item 1308 is displayed in object staginguser interface 5004. Because the input to display virtual model 1326 wasreceived in object selection user interface 1360, which is not ashopping user interface for acquiring content, no purchase button(selectable user interface object 1328) is displayed.

While virtual model 1326 is displayed in object staging user interface5004, as shown in FIG. 13I, an input (e.g., a tap input) by contact 1366is detected on touch-sensitive display system 112 at a location thatcorresponds to the “AR” region of toggle control 5008. In response tothe input, display of object staging user interface 5004 is replaced bydisplay of virtual model 1326 in an augmented reality user interface5022 (e.g., virtual model 1326 is displayed in a physical environmentcaptured by one or more cameras of device 100), as illustrated in FIG.13J. Because the input to display virtual model 1326 was received inobject selection user interface 1360, which is not a shopping userinterface for acquiring content, no purchase button (selectable userinterface object 1328) is displayed in augmented reality user interface5022.

FIGS. 14A-14B are flow diagrams illustrating method 1400 process fordisplaying a virtual model of content concurrently with a selectableuser interface object for performing an operation associated with thecontent, in accordance with some embodiments. Method 1400 is performedat an electronic device (e.g., device 300, FIG. 3, or portablemultifunction device 100, FIG. 1A) with a display generation component(e.g., a display, a projector, a heads-up display, etc.) and one or moreinput devices (e.g., a touch-screen display that serves both as thedisplay and the touch-sensitive surface, cameras, controllers,joysticks, buttons, etc.). In some embodiments, the display generationcomponent is a touch-screen display and the touch-sensitive surface isan input device that is on or integrated with the display generationcomponent. In some embodiments, the display generation component isseparate from one or more input devices. Some operations in method 1400are, optionally, combined and/or the order of some operations is,optionally, changed.

The device displays (1402), by the display generation component (e.g.,touch-sensitive display system 112), a first user interface with a firstrepresentation (e.g., a two-dimensional representation) of content(e.g., one or more content items displayed in a website or application).For example, the first user interface is shopping user interface 1300that displays content items 1302, 1304, 1306, and 1308, as describedwith regard to FIG. 13A, or an object selection user interface 1360 thatdisplays content item 1362, as described with regard to FIG. 13H. Insome embodiments, the content includes representations of one or morephysical objects (e.g., to be delivered to the user when acquired). Insome embodiments, the content includes media items, such as virtualobjects, video, images, and/or audio (e.g., to be provided to the deviceof the user when acquired). For example, content includes media item(e.g., a photograph) that is displayable in an augmented reality view asdescribed with regard to FIGS. 6A-6AI and 10A-10C. In some embodiments,the one or more content items are currently selected content items.

While displaying the first user interface with the first representationof the content, the device receives (1404), via the one or more inputdevices, a request to display a virtual model (e.g., a three-dimensionalrepresentation) that corresponds to the content (e.g., the virtual modelincludes one or more virtual objects that correspond respectively to theone or more content items). For example, the request to display avirtual model is an input at a location that corresponds to a contentitem (e.g., a tap input by contact 1324 at a location that correspondsto content item 1308, as described with regard to FIG. 13A, or a tapinput by contact 1364 at a location that corresponds to content item1362, as described with regard to FIG. 13H).

In response to receiving the request to display the virtual model thatcorresponds to the content (1406): in accordance with a determinationthat the first user interface is configured to perform an operationassociated with the content (e.g., an operation for acquiring (e.g.,downloading and/or purchasing) the content), the device displays thevirtual model of the content concurrently with a selectable userinterface object (e.g., a control for activating the operation foracquiring the content) for performing the operation associated with thecontent. For example, shopping user interface 1300, as described withregard to FIG. 13A, is configured to perform an acquisition operationfor acquiring selected content. In response to a request to display avirtual model while shopping user interface 1300 is displayed, virtualmodel 1326 is displayed concurrently with selectable user interfaceobject 1328, as described with regard to FIGS. 13B and 13C. Inaccordance with a determination that the first user interface is notconfigured to perform the operation associated with the content (e.g.,the user interface is an application or website that does not include afeature for acquiring content) the device displays the virtual model ofthe content without displaying a selectable user interface object forperforming the operation associated with the content. For example,object selection user interface 1360, as described with regard to FIG.13H, is not configured to perform the acquisition operation foracquiring selected content. In response to a request to display avirtual model while object selection user interface 1360 is displayed,virtual model 1326 is displayed without displaying selectable userinterface object 1328, as described with regard to FIGS. 131 and 13J. Insome embodiments, the user interface that is configured to perform theoperation associated with the content is a user interface of a websiteor application for acquiring content. In some embodiments, the userinterface that is not configured to perform the operation associatedwith the content is a file management interface for viewing, selecting,and/or otherwise interacting with files. In some embodiments, theoperation associated with the content is an operation for paying for thecontent and/or displaying a payment user interface (e.g., to authorizepayment and/or provide payment details) for the content. In someembodiments, the operation associated with the content is an operationfor performing an animation sequence associated with the content (e.g.,as described with regard to FIGS. 5A-5AK and 9A-9C). In someembodiments, the operation associated with the content is an operationfor sharing the content. Displaying a virtual model of contentconcurrently with a selectable user interface object in accordance witha determination that a user interface is configured to perform anoperation associated with the content (e.g., a button for acquiring thecontent) reduces the number of inputs needed to perform an operation(e.g., by allowing a user to select the button for acquiring the contentwhile viewing the virtual model rather than requiring the user toprovide additional input to access a separate interface in order toacquire the content). Reducing the number of inputs needed to perform anoperation enhances operability of the device and makes the user-deviceinterface more efficient which, additionally, reduces power usage andimproves battery life of the device by enabling the user to use thedevice more quickly and efficiently.

In some embodiments, the device detects (1408) an input directed to theselectable user interface object for performing the operation associatedwith the content; and in response to detecting the input directed to theselectable user interface object for performing the operation associatedwith the content, the device performs the operation (e.g., acquire thecontent and/or display a user interface that facilitates (e.g., displaysprompts to provide input related to) acquisition of the content). Forexample, as described with regard to FIGS. 13C-13D, an input is detectedat a location that corresponds to with selectable user interface object1328, and, in response to the input, a payment interface (e.g., nativepayment interface 1340, as described with regard to FIG. 13D, or storepayment interface 1342, as described with regard to FIG. 13E) isdisplayed. Performing an operation associated with content in responseto input directed to a selectable user interface object (e.g., a buttonfor acquiring the content) that is displayed concurrently with a virtualmodel of the content reduces the number of inputs needed to perform anoperation (e.g., by allowing a user to acquire the content while viewingof the virtual model rather than requiring the user to provideadditional input to access a separate interface in order to acquire thecontent). Reducing the number of inputs needed to perform an operationenhances operability of the device and makes the user-device interfacemore efficient which, additionally, reduces power usage and improvesbattery life of the device by enabling the user to use the device morequickly and efficiently.

In some embodiments, while displaying the virtual model of the contentconcurrently with the selectable user interface object for performingthe operation associated with the content, the device displays (1410)data that corresponds to the content (e.g., metadata for the content,such as price, length, width, height, circumference, area, volume,weight, size, available sizes, color, and/or available colors). Forexample, as described with regard to FIGS. 13C and 13D, metadata 1322 isdisplayed while virtual model 1326 is displayed concurrently withselectable user interface object 1336. In some embodiments, metadata isstored in association with the one or more content items (e.g., by anapplication that displays the first representation of the content or bya server that provides content displayed by a website or application).Displaying data that corresponds to content while displaying a virtualmodel of the content reduces the number of inputs needed to view datathat corresponds to the content (e.g., by allowing a user to viewcontent metadata while viewing the virtual model rather than requiringthe user to provide additional input to view the content metadata in aseparate interface). Reducing the number of inputs needed to perform anoperation enhances operability of the device and makes the user-deviceinterface more efficient which, additionally, reduces power usage andimproves battery life of the device by enabling the user to use thedevice more quickly and efficiently.

In some embodiments, the content includes (1412) a plurality of contentitems (e.g., content items that have been selected for acquisition) andthe virtual model of the content includes a plurality of virtualobjects, wherein a respective virtual object of the plurality of virtualobjects corresponds to a respective content item of the plurality ofcontent items. (For example, as described with regard to FIGS. 13F-13G,content that includes selected content items 1302, 1304, and 1308. Inresponse to a request to display the virtual model that corresponds tothe content, a virtual model is displayed in FIG. 13G that includesvirtual object 1356 associated with content item 1302, virtual object1358 associated with content item 1304, and virtual object 1360associated with content item 1308. Displaying a virtual model thatincludes a plurality of virtual objects, wherein a respective virtualobject of the plurality of virtual objects corresponds to a respectivecontent item of the plurality of content items, reduces the number ofinputs needed to view multiple virtual objects (e.g., by allowing a userto view multiple virtual objects concurrently without having to providemultiple inputs in order to display each virtual object individually).Reducing the number of inputs needed to perform an operation enhancesoperability of the device and makes the user-device interface moreefficient which, additionally, reduces power usage and improves batterylife of the device by enabling the user to use the device more quicklyand efficiently.

In some embodiments, the virtual model of the content and the selectableuser interface object for performing the operation associated with thecontent are displayed (1414) in the first user interface (e.g., inshopping user interface 1300). Displaying the virtual model of thecontent and the selectable user interface object for performing theoperation associated with the content (e.g., a button for acquiring thecontent) in the first user interface displayed while the request todisplay the virtual model is received reduces the number of inputsrequired to perform an operation (e.g., by allowing a user to select thebutton for acquiring the content from while viewing the virtual model inthe first user interface, rather than requiring the user to provideadditional input to access a separate interface in order to acquire thecontent). Reducing the number of inputs needed to perform an operationenhances operability of the device and makes the user-device interfacemore efficient which, additionally, reduces power usage and improvesbattery life of the device by enabling the user to use the device morequickly and efficiently.

In some embodiments, displaying the virtual model of the contentconcurrently with the selectable user interface object includes (1416):displaying a second user interface (e.g., replacing display of the firstuser interface and/or at least partially overlaying the first userinterface) and displaying the virtual model of the content concurrentlywith the selectable user interface object in the second user interface(e.g., an interstitial user interface, an augmented reality userinterface, a virtual environment, and/or a user interface for providingacquisition authorization input). For example, the virtual model (e.g.,virtual model 1326) is displayed concurrently with the selectable userinterface object (e.g., selectable user interface object 1328) in anobject staging user interface 5004, as described with regard to FIG.13B, or in an augmented reality user interface 5022, as described withregard to FIG. 13C. Displaying the virtual model of the content and theselectable user interface object for performing the operation associatedwith the content (e.g., a button for acquiring the content) in a seconduser interface without requiring further user input enhances theoperability of the device (e.g., by displaying a second user interfaceand displaying the button in response to a single input instead ofrequiring separate inputs to display the second user interface and todisplay the button) and makes the user-device interface more efficientwhich, additionally, reduces power usage and improves battery life ofthe device by enabling the user to use the device more quickly andefficiently.

In some embodiments, while displaying the second user interface (e.g., astaging user interface that includes a simulated three-dimensional spacein which a three-dimensional representation of the virtual object may bemanipulated (e.g., rotated around an x-axis, a y-axis, and/or a z-axis,and/or resized) in response to user input), the device detects (1418) amanipulation input directed to the virtual model (e.g., a pinch or swipeat a location on a touch-sensitive surface that corresponds to thevirtual model or that corresponds to a respective virtual object of aplurality of virtual objects of the model) and, in response to detectingthe manipulation input directed to the virtual model, the device changesa display property of the virtual model based on the manipulation input(e.g., rotating and/or resizing the virtual model (e.g., collectively orindividually rotating and/or resizing one or more virtual objects)). Forexample, while the virtual model (e.g., virtual model 1326) is displayedconcurrently with the selectable user interface object (e.g., selectableuser interface object 1328) in an object staging user interface 5004, asdescribed with regard to FIG. 13B, a manipulation input is received formanipulating virtual model 1326. Manipulation of virtual objects in anobject staging user interface 5004 is described further with regard toFIGS. 5B-5C. In some embodiments, while the second user interface isdisplayed, an input for displaying the virtual model in an augmentedreality view is detected (e.g., at a location of a displayed control fordisplaying an augmented reality view). In response to detecting theinput for displaying the virtual model in the augmented reality view,the virtual model is displayed with at least a portion of a field ofview of the one or more cameras. For example, in response to an inputdetected at a location that corresponds to the “AR” region of togglecontrol 5008, display of virtual model 1326 in object staging userinterface 5004 is replaced with display of virtual model 1326 with atleast a portion of a field of view of one or more cameras in anaugmented reality user interface 5022, as described with regard to FIGS.13B-13C. Changing a display property of a virtual model based on amanipulation input directed to the virtual model provides improvedvisual feedback to the user (e.g., indicating that the virtual model ischangeable in response to user input). Providing improved visualfeedback to the user enhances the operability of the device and makesthe user-device interface more efficient (e.g., by helping the user toprovide proper inputs and reducing user mistakes whenoperating/interacting with the device) which, additionally, reducespower usage and improves battery life of the device by enabling the userto use the device more quickly and efficiently.

In some embodiments, the electronic device includes (1420) one or morecameras (e.g., one or more rear-facing cameras on a side of the deviceopposite from the display and the touch-sensitive surface) anddisplaying the second user interface includes displaying the virtualmodel and the selectable user interface object with at least a portionof the field of view of the one or more cameras. For example, thevirtual model (e.g., virtual model 1326) is displayed concurrently withthe selectable user interface object (e.g., selectable user interfaceobject 1328) in an augmented reality user interface 5022, as describedwith regard to FIG. 13C. Displaying the virtual model of the content andthe selectable user interface object for performing the operationassociated with the content (e.g., a button for acquiring the content)with at least a portion of the field of view of the one or more cameras(e.g., displaying the virtual model in an augmented reality view of aphysical environment) without requiring further user input enhances theoperability of the device (e.g., by displaying the virtual model in theaugmented reality view and displaying the button for acquiring thecontent in response to a single input rather than requiring separateinputs to display the virtual model in the augmented reality view and todisplay the button for acquiring the content) and makes the user-deviceinterface more efficient which, additionally, reduces power usage andimproves battery life of the device by enabling the user to use thedevice more quickly and efficiently.

In some embodiments, the selectable user interface object for performingthe operation associated with the content is displayed (1422) at alocation that corresponds to the displayed portion of the field of viewof the one or more cameras. For example, as described with regard toFIG. 13C, the selectable user interface object (e.g., selectable userinterface object 1328) is displayed overlaying the displayed portion ofthe field of view of the one or more cameras (e.g., as shown inaugmented reality user interface 5022). Displaying the selectable userinterface object for performing the operation associated with thecontent (e.g., a button for acquiring the content) at a location thatcorresponds to the displayed portion of the field of view of the one ormore cameras reduces the number of inputs required to perform anoperation (e.g., by displaying the virtual model in the augmentedreality view and displaying the button for acquiring the content inresponse to a single input rather than requiring separate inputs todisplay the virtual model in the augmented reality view and to displaythe button for acquiring the content). Reducing the number of inputsrequired to perform an operation enhances the operability of the deviceand makes the user-device interface more efficient which, additionally,reduces power usage and improves battery life of the device by enablingthe user to use the device more quickly and efficiently.

In some embodiments, the selectable user interface object for performingthe operation associated with the content has (1424) a display property(e.g., at least a portion of the selectable user interface object istranslucent) that renders the displayed portion of the field of view ofthe one or more cameras partially visible through the selectable userinterface object. For example, in FIG. 13C, the floor of the physicalenvironment visible in augmented reality user interface 5022 is partialvisible through selectable user interface object 1328. Displaying theselectable user interface object for performing the operation associatedwith the content (e.g., a button for acquiring the content) with adisplay property that allows the displayed portion of the field of viewof the one or more cameras partially visible through the selectable userinterface object (e.g., displaying a translucent button through whichthe augmented reality view of the physical environment and/or thevirtual model is visible) reduces the number of inputs required toperform the operation (e.g., by allowing the user to simultaneously viewthe button and the augmented reality view rather than requiring the userto provide additional input to access a separate interface in order todisplay the button for acquiring the content). Reducing the number ofinputs needed to perform an operation enhances the operability of thedevice and makes the user-device interface more efficient which,additionally, reduces power usage and improves battery life of thedevice by enabling the user to use the device more quickly andefficiently.

In some embodiments, while displaying the virtual model of the contentconcurrently with the selectable user interface object for performingthe operation associated with the content, the device detects (1424) aninput directed to the selectable user interface object, and in responseto detecting the input directed to the selectable user interface object,the device displays a third user interface (e.g., replacing display ofthe first user interface or the second user interface with the thirduser interface, or displaying the third user interface at leastpartially overlaying the first user interface or the second userinterface). (For example, as described with regard FIG. 13C, an input(e.g., a tap input by contact 1336) is detected at a location thatcorresponds to selectable user interface object 1328. In response toreceiving the input, native payment interface 1340 e.g., an Apple Paytransaction processing interface) is displayed, as described with regardto FIG. 13D, or a store payment interface 1342 (e.g., a checkout pageassociated with a store that provides an application or website) isdisplayed, as described with regard to FIG. 13E. While displaying thethird user interface, the device detects an input to authorizeacquisition of the content (e.g., an input at a sensor for fingerprint,thumbprint, or facial recognition, input at a displayed number pad entryprompt, input at a physical button of the device, and/or an input at alocation on a touchscreen that corresponds to a displayed authorizationuser interface object). For example, the input to authorize acquisitionof the content is a thumbprint input detected by a thumbprint sensor ata location indicated by region 1341, as described with regard to FIG.13D, or the input to authorize acquisition of the content is an input(e.g., a tap input) at a location that corresponds to “Place Order”button 1344), as described with regard to FIG. 13E. In response todetecting the request to authorize the acquisition of the content, thedevice acquires the content (e.g., downloading the content and/ortransmitting authorization information to the application, the website,and/or a transaction processing service for acquiring the content).Displaying a third user interface for authorizing acquisition of contentin response to detecting an input detected while displaying the virtualmodel is displayed reduces a number of inputs required (e.g., bydisplaying the interface for authorizing acquisition of content directlyfrom the interface that displays the virtual model, rather thanrequiring the user to provide additional input to navigate away from theinterface that displays the virtual model to a separate interface thatdisplays a control for displaying the interface for authorizingacquisition of content) and makes the user-device interface moreefficient which, additionally, reduces power usage and improves batterylife of the device by enabling the user to use the device more quicklyand efficiently.

It should be understood that the particular order in which theoperations in FIGS. 14A-14B have been described is merely an example andis not intended to indicate that the described order is the only orderin which the operations could be performed. One of ordinary skill in theart would recognize various ways to reorder the operations describedherein. Additionally, it should be noted that details of other processesdescribed herein with respect to other methods described herein (e.g.,methods 900, 1000, 1100, and 1200) are also applicable in an analogousmanner to method 1400 described above with respect to FIGS. 14A-14B. Forexample, the contacts, inputs, gestures, virtual model, virtual objects,environments, user interface regions, user interface objects, and/orfields of view described above with reference to method 1400 optionallyhave one or more of the characteristics of the contacts, inputs,gestures, media items, virtual objects, environments, user interfaceregions, user interface objects, and/or fields of view described hereinwith reference to other methods described herein (e.g. methods 900,1000, 1100, and 1200). For brevity, these details are not repeated here.

The operations described above with reference to 9A-9C, 10A-10C,11A-11D, 12A-12B, and 14A-14B are, optionally, implemented by componentsdepicted in FIGS. 1A-1B. For example, display operations 902, 1002,1006, 1010, 1102, 1202, 1206, 1402, and 1406; detection operations 904,1004, 1008, 1104, 1204; performance operation 906; change operation1106, and receiving operation 1404 are, optionally, implemented by eventsorter 170, event recognizer 180, and event handler 190. Event monitor171 in event sorter 170 detects a contact on touch-sensitive displaysystem 112, and event dispatcher module 174 delivers the eventinformation to application 136-1. A respective event recognizer 180 ofapplication 136-1 compares the event information to respective eventdefinitions 186, and determines whether a first contact at a firstlocation on the touch-sensitive surface (or whether rotation of thedevice) corresponds to a predefined event or sub-event, such asselection of an object on a user interface, or rotation of the devicefrom one orientation to another. When a respective predefined event orsub-event is detected, event recognizer 180 activates an event handler190 associated with the detection of the event or sub-event. Eventhandler 190 optionally uses or calls data updater 176 or object updater177 to update the application internal state 192. In some embodiments,event handler 190 accesses a respective GUI updater 178 to update whatis displayed by the application. Similarly, it would be clear to aperson having ordinary skill in the art how other processes can beimplemented based on the components depicted in FIGS. 1A-1B.

The foregoing description, for purpose of explanation, has beendescribed with reference to specific embodiments. However, theillustrative discussions above are not intended to be exhaustive or tolimit the invention to the precise forms disclosed. Many modificationsand variations are possible in view of the above teachings. Theembodiments were chosen and described in order to best explain theprinciples of the invention and its practical applications, to therebyenable others skilled in the art to best use the invention and variousdescribed embodiments with various modifications as are suited to theparticular use contemplated.

1. A method, comprising: at an electronic device including a displaygeneration component and one or more input devices: displaying, via thedisplay generation component, at least a portion of an environment thatincludes a virtual object that is associated with a first action that istriggered based on satisfaction of a first set of criteria; whiledisplaying, by the display generation component, the portion of theenvironment, detecting a first user input; and in response to detectingthe first user input: in accordance with a determination that the firstuser input satisfies the first set of criteria, performing the firstaction; and in accordance with a determination that the first user inputdoes not satisfy the first set of criteria but instead satisfies asecond set of criteria, forgoing performing the first action and insteaddisplaying a first visual indication of one or more user inputs that ifperformed would cause the first set of criteria to be satisfied, whereinthe first visual indication is displayed as a user interface object thatis distinct from the virtual object.
 2. The method of claim 1, whereinthe first set of criteria include criteria that are satisfied inaccordance with a determination that at least a portion of theelectronic device moves to a position that is within a defined distanceof a displayed position of the virtual object.
 3. The method of claim 1,wherein the one or more input devices include a touch-screen display andthe first set of criteria include criteria that are satisfied inaccordance with a determination that the first user input is a tap inputby a contact on the touch-screen display at a location that correspondsto the virtual object.
 4. The method of claim 1, wherein the one or moreinput devices include a sensor for detecting a direction of a gaze of auser of the electronic device and the first set of criteria includecriteria that are satisfied in accordance with a determination that thegaze of the user is directed toward a location that corresponds to thevirtual object.
 5. The method of claim 1, wherein: the electronic deviceincludes one or more cameras; the displayed environment includes atleast a portion of the field of view of the one or more cameras; and themethod includes: detecting movement of the electronic device thatadjusts the field of view of the one or more cameras and, in response todetecting the movement of the electronic device, adjusting the virtualobject in accordance with a fixed spatial relationship between thevirtual object and a respective plane in the field of view of the one ormore cameras.
 6. The method of claim 5, wherein the first set ofcriteria include criteria that are satisfied in accordance with adetermination that the virtual object is displayed within a designatedportion of the field of view of the one or more cameras.
 7. The methodof claim 5, wherein the second set of criteria include criteria that aremet in accordance with a determination that the first user input is aninput for moving the virtual object in the environment.
 8. The method ofclaim 5, including: receiving a request to display the virtual object ata first location in the environment; and in response to the request todisplay the virtual object at the first location in the environment: inaccordance with a determination that a surface size that corresponds tothe first location does not meet size compatibility criteria for thevirtual object, displaying a visual indication that provides informationindicating that the first location does not meet the size compatibilitycriteria; and in accordance with a determination that the surface sizethat corresponds to the first location meets the size compatibilitycriteria for the virtual object, displaying the virtual object at thefirst location in the environment.
 9. The method of claim 5, wherein:the movement of the electronic device is detected while performing thefirst action; and adjusting the virtual object in response to detectingthe movement of the electronic device in accordance with the fixedspatial relationship between the virtual object and the respective planecauses the virtual object to move at least partially beyond thedisplayed portion of the field of view of the one or more cameras; andthe method includes: while the virtual object is at least partiallybeyond the displayed portion of the field of view of the one or morecameras, displaying a second visual indication of one or more inputsthat if performed would cause the first set of criteria to be satisfied.10. The method of claim 1, wherein: the electronic device includes oneor more cameras; and the method includes, in accordance with adetermination that the first user input corresponds to a request todisplay the virtual object in an augmented reality environment,replacing display of the environment that includes the virtual objectwith display of at least a portion of the field of view of the one ormore cameras.
 11. The method of claim 10, wherein the second set ofcriteria include criteria that are met in accordance with adetermination that the first user input is an input for altering acharacteristic of the virtual object.
 12. The method of claim 1, whereinthe first visual indication of the one or more inputs that if performedwould cause the first set of criteria to be satisfied includes a textprompt.
 13. The method of claim 1, including: while displaying the firstvisual indication of the one or more inputs that if performed wouldcause the first set of criteria to be satisfied, detecting a secondinput that satisfies the first set of criteria; and in response todetecting the second input that satisfies the first set of criteria,ceasing to display the first visual indication.
 14. The method of claim1, wherein the electronic device includes one or more cameras; thedisplayed environment includes at least a portion of the field of viewof the one or more cameras, and the second set of criteria includecriteria that are met in accordance with a determination that the firstuser input is an input for moving the virtual object in the environment.15. The method of claim 1, wherein performing the first action includesanimating at least a portion of the virtual object, and the methodincludes displaying a video play head control for adjusting playback ofan animated sequence that animates the portion of the virtual object.16. The method of claim 1, wherein: performing the first action includesanimating at least a portion of the virtual object; the electronicdevice includes an audio generation component; performing the firstaction includes generating, by the audio generation component, a firstaudio channel and a second audio channel; and the method includesadjusting an amplitude of the first audio channel and an amplitude ofthe second audio channel in accordance with a displayed position of thevirtual object in the environment.
 17. A computer system, comprising: adisplay generation component; one or more input devices; one or moreprocessors; and memory storing one or more programs, wherein the one ormore programs are configured to be executed by the one or moreprocessors, the one or more programs including instructions for:displaying, via the display generation component, at least a portion ofan environment that includes a virtual object that is associated with afirst action that is triggered based on satisfaction of a first set ofcriteria; while displaying, by the display generation component, theportion of the environment, detecting a first user input; and inresponse to detecting the first user input: in accordance with adetermination that the first user input satisfies the first set ofcriteria, performing the first action; and in accordance with adetermination that the first user input does not satisfy the first setof criteria but instead satisfies a second set of criteria, forgoingperforming the first action and instead displaying a first visualindication of one or more user inputs that if performed would cause thefirst set of criteria to be satisfied, wherein the first visualindication is displayed as a user interface object that is distinct fromthe virtual object.
 18. A non-transitory computer readable storagemedium storing one or more programs, the one or more programs comprisinginstructions, which, when executed by a computer system with a displaygeneration component and one or more input devices, cause the computersystem to: display, via the display generation component, at least aportion of an environment that includes a virtual object that isassociated with a first action that is triggered based on satisfactionof a first set of criteria; while displaying, by the display generationcomponent, the portion of the environment, detect a first user input;and in response to detecting the first user input: in accordance with adetermination that the first user input satisfies the first set ofcriteria, perform the first action; and in accordance with adetermination that the first user input does not satisfy the first setof criteria but instead satisfies a second set of criteria, forgoperforming the first action and instead display a first visualindication of one or more user inputs that if performed would cause thefirst set of criteria to be satisfied, wherein the first visualindication is displayed as a user interface object that is distinct fromthe virtual object.
 19. The non-transitory computer readable storagemedium of claim 18, wherein the first set of criteria include criteriathat are satisfied in accordance with a determination that at least aportion of the electronic device moves to a position that is within adefined distance of a displayed position of the virtual object.
 20. Thenon-transitory computer readable storage medium of claim 18, wherein theone or more input devices include a touch-screen display and the firstset of criteria include criteria that are satisfied in accordance with adetermination that the first user input is a tap input by a contact onthe touch-screen display at a location that corresponds to the virtualobject.